nohup: ignoring input
I0120 09:32:22.732915 140191300101952 af.py:89] Mounting /home/goddard/ucsf -> /mnt/fasta_path_0
I0120 09:32:22.732996 140191300101952 af.py:89] Mounting /home/goddard/ucsf/databases/alphafold21/uniref90 -> /mnt/uniref90_database_path
I0120 09:32:22.733039 140191300101952 af.py:89] Mounting /home/goddard/ucsf/databases/alphafold21/mgnify -> /mnt/mgnify_database_path
I0120 09:32:22.733076 140191300101952 af.py:89] Mounting /home/goddard/ucsf/databases -> /mnt/data_dir
I0120 09:32:22.733111 140191300101952 af.py:89] Mounting /home/goddard/ucsf/databases/alphafold21/pdb_mmcif -> /mnt/template_mmcif_dir
I0120 09:32:22.733144 140191300101952 af.py:89] Mounting /home/goddard/ucsf/databases/alphafold21/pdb_mmcif -> /mnt/obsolete_pdbs_path
I0120 09:32:22.733178 140191300101952 af.py:89] Mounting /home/goddard/ucsf/databases/alphafold21/pdb70 -> /mnt/pdb70_database_path
I0120 09:32:22.733210 140191300101952 af.py:89] Mounting /home/goddard/ucsf/databases/alphafold21/uniclust30/uniclust30_2018_08 -> /mnt/uniclust30_database_path
I0120 09:32:22.733253 140191300101952 af.py:89] Mounting /home/goddard/ucsf/databases/alphafold21/bfd -> /mnt/bfd_database_path
I0120 09:32:29.271636 140191300101952 af.py:224] I0120 17:32:29.269736 139969096478912 templates.py:857] Using precomputed obsolete pdbs /mnt/obsolete_pdbs_path/obsolete.dat.
I0120 09:32:29.997471 140191300101952 af.py:224] I0120 17:32:29.997162 139969096478912 xla_bridge.py:244] Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker:
I0120 09:32:30.171287 140191300101952 af.py:224] I0120 17:32:30.170579 139969096478912 xla_bridge.py:244] Unable to initialize backend 'tpu': INVALID_ARGUMENT: TpuPlatform is not available.
I0120 09:32:35.695491 140191300101952 af.py:224] I0120 17:32:35.695006 139969096478912 run_alphafold.py:385] Have 5 models: ['model_1', 'model_2', 'model_3', 'model_4', 'model_5']
I0120 09:32:35.695607 140191300101952 af.py:224] I0120 17:32:35.695113 139969096478912 run_alphafold.py:397] Using random seed 869311309730577466 for the data pipeline
I0120 09:32:35.695659 140191300101952 af.py:224] I0120 17:32:35.695246 139969096478912 run_alphafold.py:150] Predicting seq_6um1
I0120 09:32:35.699694 140191300101952 af.py:224] I0120 17:32:35.699160 139969096478912 jackhmmer.py:130] Launching subprocess "/usr/bin/jackhmmer -o /dev/null -A /tmp/tmpdk90o9sp/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /mnt/fasta_path_0/seq_6um1.fasta /mnt/uniref90_database_path/uniref90.fasta"
I0120 09:32:35.740516 140191300101952 af.py:224] I0120 17:32:35.740054 139969096478912 utils.py:36] Started Jackhmmer (uniref90.fasta) query
I0120 09:37:15.566330 140191300101952 af.py:224] I0120 17:37:15.565832 139969096478912 utils.py:40] Finished Jackhmmer (uniref90.fasta) query in 279.826 seconds
I0120 09:37:15.694481 140191300101952 af.py:224] I0120 17:37:15.694036 139969096478912 jackhmmer.py:130] Launching subprocess "/usr/bin/jackhmmer -o /dev/null -A /tmp/tmpwxj677vc/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /mnt/fasta_path_0/seq_6um1.fasta /mnt/mgnify_database_path/mgy_clusters_2018_12.fa"
I0120 09:37:15.737852 140191300101952 af.py:224] I0120 17:37:15.736943 139969096478912 utils.py:36] Started Jackhmmer (mgy_clusters_2018_12.fa) query
I0120 09:43:03.018820 140191300101952 af.py:224] I0120 17:43:03.018344 139969096478912 utils.py:40] Finished Jackhmmer (mgy_clusters_2018_12.fa) query in 347.281 seconds
I0120 09:43:08.518623 140191300101952 af.py:224] I0120 17:43:08.518146 139969096478912 hhsearch.py:85] Launching subprocess "/usr/bin/hhsearch -i /tmp/tmpll4r4jyb/query.a3m -o /tmp/tmpll4r4jyb/output.hhr -maxseq 1000000 -d /mnt/pdb70_database_path/pdb70"
I0120 09:43:08.562161 140191300101952 af.py:224] I0120 17:43:08.561176 139969096478912 utils.py:36] Started HHsearch query
I0120 09:45:06.214850 140191300101952 af.py:224] I0120 17:45:06.214344 139969096478912 utils.py:40] Finished HHsearch query in 117.653 seconds
I0120 09:45:10.726975 140191300101952 af.py:224] I0120 17:45:10.726543 139969096478912 hhblits.py:128] Launching subprocess "/usr/bin/hhblits -i /mnt/fasta_path_0/seq_6um1.fasta -cpu 4 -oa3m /tmp/tmp288cpamt/output.a3m -o /dev/null -n 3 -e 0.001 -maxseq 1000000 -realign_max 100000 -maxfilt 100000 -min_prefilter_hits 1000 -d /mnt/bfd_database_path/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt -d /mnt/uniclust30_database_path/uniclust30_2018_08"
I0120 09:45:10.772521 140191300101952 af.py:224] I0120 17:45:10.771992 139969096478912 utils.py:36] Started HHblits query
I0120 10:27:38.874136 140191300101952 af.py:224] I0120 18:27:38.873388 139969096478912 utils.py:40] Finished HHblits query in 2548.101 seconds
I0120 10:27:41.887820 140191300101952 af.py:224] I0120 18:27:41.886935 139969096478912 templates.py:878] Searching for template for: MEAAAGRSSHLGPAPAGRPPRCPLLLQLQLLLLLLLLPPGWVPGAAGTQGAEFPELCSYTWEAVDTKNNMLYKINICGNMGVAQCGPSSAVCMHDLKTDSFHSVGDSLLKTASRSLLEFNTTVNCKQQNHKIQSSITFLCGKTLGTPEFVTATDCVHYFEWRTTAACKKNIFKANKEVPCYAFDRELKKHDLNPLIKTSGAYLVDDSDPDTSLFINVCRDIEVLRASSPQVRVCPTGAAACLVRGDRAFDVGRPQEGLKLVSNDRLVLSYVKEGAGQPDFCDGHSPAVTITFVCPSERREGTIPKLTAKSNCRFEIEWVTEYACHRDYLESRSCSLSSAQHDVAVDLQPLSRVEASDSLFYTSEADEYTYYLSICGGSQAPICNKKDAAVCQVKKADSTQVKVAGRPQNLTLRYSDGDLTLIYFGGEECSSGFQRMSVINFECNQTAGNNGRGAPVFTGEVDCTYFFTWDTKYACVHEKEALLCGVSDGKQRFDLSALARHSELEQNWEAVDGSQREAEKKHFFINICHRVLQTGQARGCPEDAAVCAVDKNGSKNLGRFISSPTREKGNIQLSYSDGDECGGGQKIITNITLMCKPGDLESAPVLTTSRADGCFYEFEWRTAAACVLSRTEGDNCTVFDSQAGFSFDLTPLTKKDAYKVETDKYEFHINVCGPVSVGACPPDSGACQVSRSDRKSWNLGRSNAKLSYYDGMIQLTYRDGTPYNNEKRTPRATLITFLCDRDAGVGFPEYQEEDNSTYNFRWYTSYACPEEPLECIVTDPVTLDQYDLSRLAKSEGGPGGNWYSLDNGGARSTWRKYYINVCRPLNPVPGCDRYASACQMKYQGEQGSYSETVSISNLGVAKTGPMVEDSGSLLLEYVNGSACTTSDQRRTTYTTRIHLVCSTGSLYTHPIFSLNWECVVSFLWNTAAACPIRITTDIDQVCSIKDPNSGYVFDLNPLNNSRGYVVLGIGKTFLFNVCGDMPACGTLDGKPASGCEAEVQMDDMKTLKPGRLVGLEKSLQLSTEGFITLNYTGLPSHPNGRADAFIIRFVCNDDVYPGTPKFLHQDIDSSLGIRDTFFEFETALACVPSPVDCQVTDPAGNEYDLSGLSKARKPWTAVDTFDEGKKRTFYLSVCTPLPYIPGCHGTAVGCCLVTEDSKLNLGVVQISPQVGANGSLSLVYVNGDKCKNQRFSTRINLECAHTTGSPTFQLQNDCEYVFLWRTVEACPVVRAEGDYCEVRDPRHGNLYNLIPLGLNDTVVRAGEYTYYFRVCGELTSGVCPTSDKSKVISSCQEKRGPQGFQKVAGLFNQKLTYENGVLKMNYTGGDTCHKVYQRSTTIFFYCDRSTQAPVFLQETSDCSYLFEWRTQYACPPYDLTECSFKNEAGETYDLSSLSRYSDNWEAVTGTGSTEHYLINVCKSLSPQAGSDPCPPEAAVCLLGGPKPVNLGRVRDSPQWSQGLTLLKYVDGDLCPDQIRKKSTTIRFTCSESHVNSRPMFISAVEDCEYTFSWPTAAACAVKSNVHDDCQVTNPATGHLFDLSSLSGRAGFTAAYSEKGLVYLSVCGDNENCANGVGACFGQTRISVGKASKRLTYVDQVLQLVYEGGSPCPSKTGLSYKSVISFVCRPEVGPTNRPMLISLDKRTCTLFFSWHTPLACEQTTECSVRNGSSLIDLSPLIHRTGGYEAYDESEDDGSDTSPDFYINICQPLNPMHGLACPAGTAVCKVPVDGPPIDIGRVAGPPILNPIANEVYLNFESSTPCLADRHFNYTSLITFHCKRGVSMGTPKLLRTSVCDFVFEWETPLVCPDEVKTDGCSLTDEQLYYSFNLSSLSKSTFKVTRGPHTYSVGVCTAAAGLDEGGCKDGAVCLLSGSKGASFGRLASMKLDYRHQDEAVILSYANGDTCPPETEDGEPCVFPFVFNGKSYEECVVESRARLWCATTANYDRDHEWGFCKHSTSHRTSVIIFKCDEDADVGRPQVFSEVRGCEVTFEWKTKVVCPPKKMECKFVQKHRTYDLRLLSSLTGSWSFVHNGASYYINLCQKIYKGPQDCSERASVCKKSTSGEVQVLGLVHTQKLDVVDDRVIVTYSKGHYCGDNKTASAVIELTCAKTVGRPSFTRFDVDSCTYHFSWDSRAACAVKPQEVQMVNGTITNPANGRSFSLGDIYFKRFSASGDVRTNGDRYIYEIQLSSITGSSSPACSGASICQRKANDQHFSRKVGTSNQTRYYVQDGDLDVVFTSSSKCGKDKTKSVSSTIFFHCDPLVKDGIPEFSHETADCQYLFSWHTSAVCPLGAGFDEEIAGDDAQEHKGLSERSQAVGAVLSLLLVALTACLLTLLLYKKERREMVMSRLTNCCRRSANVSYKYSKVNKEEEADENETEWLMEEIQPPAPRPGKEGQENGHVAAKSVRAADTLSALHGDEQDSEDEVLTLPEVKVRPPGRAPGAEGGPPLRPLPRKAPPPLRADDRVGLVRGEPARRGRPRAAATPISTFHDDSDEDLLHV
I0120 10:27:41.887928 140191300101952 af.py:224] I0120 18:27:41.887391 139969096478912 templates.py:718] hit 6um1_A did not pass prefilter: Template is an exact subsequence of query with large coverage. Length ratio: 1.0.
I0120 10:27:41.887982 140191300101952 af.py:224] I0120 18:27:41.887448 139969096478912 templates.py:913] Skipped invalid hit 6UM1_A Cation-independent mannose-6-phosphate receptor; M-6-P, IGFII, receptor, SUGAR BINDING; HET: NAG, BMA; 3.46A {Bos taurus}, error: None, warning: None
I0120 10:27:42.737264 140191300101952 af.py:224] I0120 18:27:42.736808 139969096478912 templates.py:268] Found an exact template match 6um1_A.
I0120 10:27:42.805850 140191300101952 af.py:224] I0120 18:27:42.805481 139969096478912 templates.py:268] Found an exact template match 6um1_A.
I0120 10:27:42.873347 140191300101952 af.py:224] I0120 18:27:42.873005 139969096478912 templates.py:268] Found an exact template match 6um1_A.
I0120 10:27:43.118655 140191300101952 af.py:224] I0120 18:27:43.118164 139969096478912 templates.py:268] Found an exact template match 2v5o_A.
I0120 10:27:43.140908 140191300101952 af.py:224] I0120 18:27:43.140531 139969096478912 templates.py:268] Found an exact template match 2v5o_A.
I0120 10:27:43.162713 140191300101952 af.py:224] I0120 18:27:43.162437 139969096478912 templates.py:268] Found an exact template match 2v5o_A.
I0120 10:27:43.184335 140191300101952 af.py:224] I0120 18:27:43.184000 139969096478912 templates.py:268] Found an exact template match 2v5o_A.
I0120 10:27:43.685614 140191300101952 af.py:224] I0120 18:27:43.685173 139969096478912 templates.py:268] Found an exact template match 2v5p_B.
I0120 10:27:43.848972 140191300101952 af.py:224] I0120 18:27:43.848488 139969096478912 templates.py:268] Found an exact template match 1q25_A.
I0120 10:27:43.865390 140191300101952 af.py:224] I0120 18:27:43.865053 139969096478912 templates.py:268] Found an exact template match 2v5p_B.
I0120 10:27:43.883302 140191300101952 af.py:224] I0120 18:27:43.883038 139969096478912 templates.py:268] Found an exact template match 2v5p_B.
I0120 10:27:43.901421 140191300101952 af.py:224] I0120 18:27:43.901156 139969096478912 templates.py:268] Found an exact template match 2v5p_B.
I0120 10:27:43.919111 140191300101952 af.py:224] I0120 18:27:43.918804 139969096478912 templates.py:268] Found an exact template match 1q25_A.
I0120 10:27:43.936026 140191300101952 af.py:224] I0120 18:27:43.935762 139969096478912 templates.py:268] Found an exact template match 1q25_A.
I0120 10:27:43.952876 140191300101952 af.py:224] I0120 18:27:43.952613 139969096478912 templates.py:268] Found an exact template match 1q25_A.
I0120 10:27:44.054512 140191300101952 af.py:224] I0120 18:27:44.054077 139969096478912 templates.py:268] Found an exact template match 2v5n_A.
I0120 10:27:44.066761 140191300101952 af.py:224] I0120 18:27:44.066469 139969096478912 templates.py:268] Found an exact template match 2v5n_A.
I0120 10:27:44.079108 140191300101952 af.py:224] I0120 18:27:44.078812 139969096478912 templates.py:268] Found an exact template match 2v5n_A.
I0120 10:27:44.091434 140191300101952 af.py:224] I0120 18:27:44.091170 139969096478912 templates.py:268] Found an exact template match 2v5n_A.
I0120 10:27:44.103690 140191300101952 af.py:224] I0120 18:27:44.103373 139969096478912 templates.py:718] hit 2kvb_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.058823529411764705.
I0120 10:27:44.103775 140191300101952 af.py:224] I0120 18:27:44.103442 139969096478912 templates.py:913] Skipped invalid hit 2KVB_A Cation-independent mannose-6-phosphate receptor; TRANSPORT, LYSOSOME, MANNOSE, RECEPTOR, SUGAR; NMR {Bos taurus}, error: None, warning: None
I0120 10:27:44.103853 140191300101952 af.py:224] I0120 18:27:44.103497 139969096478912 templates.py:718] hit 2kvb_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05802320928371348.
I0120 10:27:44.103978 140191300101952 af.py:224] I0120 18:27:44.103539 139969096478912 templates.py:913] Skipped invalid hit 2KVB_A Cation-independent mannose-6-phosphate receptor; TRANSPORT, LYSOSOME, MANNOSE, RECEPTOR, SUGAR; NMR {Bos taurus}, error: None, warning: None
I0120 10:27:44.104063 140191300101952 af.py:224] I0120 18:27:44.103610 139969096478912 templates.py:718] hit 2kvb_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.056422569027611044.
I0120 10:27:44.104109 140191300101952 af.py:224] I0120 18:27:44.103702 139969096478912 templates.py:913] Skipped invalid hit 2KVB_A Cation-independent mannose-6-phosphate receptor; TRANSPORT, LYSOSOME, MANNOSE, RECEPTOR, SUGAR; NMR {Bos taurus}, error: None, warning: None
I0120 10:27:44.104201 140191300101952 af.py:224] I0120 18:27:44.103746 139969096478912 templates.py:718] hit 2kvb_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05682272909163665.
I0120 10:27:44.104293 140191300101952 af.py:224] I0120 18:27:44.103782 139969096478912 templates.py:913] Skipped invalid hit 2KVB_A Cation-independent mannose-6-phosphate receptor; TRANSPORT, LYSOSOME, MANNOSE, RECEPTOR, SUGAR; NMR {Bos taurus}, error: None, warning: None
I0120 10:27:44.104382 140191300101952 af.py:224] I0120 18:27:44.103851 139969096478912 templates.py:718] hit 2l21_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.055222088835534214.
I0120 10:27:44.104426 140191300101952 af.py:224] I0120 18:27:44.103938 139969096478912 templates.py:913] Skipped invalid hit 2L21_A Cation-independent mannose-6-phosphate receptor; Genomic imprinting, Insulin-like growth factor; NMR {Gallus gallus}, error: None, warning: None
I0120 10:27:44.104470 140191300101952 af.py:224] I0120 18:27:44.104021 139969096478912 templates.py:718] hit 2l2g_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.055222088835534214.
I0120 10:27:44.104514 140191300101952 af.py:224] I0120 18:27:44.104079 139969096478912 templates.py:913] Skipped invalid hit 2L2G_A entity; Insulin-like growth factor 2, mannose; NMR {Monodelphis domestica}, error: None, warning: None
I0120 10:27:44.104593 140191300101952 af.py:224] I0120 18:27:44.104129 139969096478912 templates.py:718] hit 2l21_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.055222088835534214.
I0120 10:27:44.104689 140191300101952 af.py:224] I0120 18:27:44.104226 139969096478912 templates.py:913] Skipped invalid hit 2L21_A Cation-independent mannose-6-phosphate receptor; Genomic imprinting, Insulin-like growth factor; NMR {Gallus gallus}, error: None, warning: None
I0120 10:27:44.104770 140191300101952 af.py:224] I0120 18:27:44.104293 139969096478912 templates.py:718] hit 2l2g_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05402160864345738.
I0120 10:27:44.104814 140191300101952 af.py:224] I0120 18:27:44.104347 139969096478912 templates.py:913] Skipped invalid hit 2L2G_A entity; Insulin-like growth factor 2, mannose; NMR {Monodelphis domestica}, error: None, warning: None
I0120 10:27:44.104858 140191300101952 af.py:224] I0120 18:27:44.104416 139969096478912 templates.py:718] hit 2l21_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05322128851540616.
I0120 10:27:44.104901 140191300101952 af.py:224] I0120 18:27:44.104451 139969096478912 templates.py:913] Skipped invalid hit 2L21_A Cation-independent mannose-6-phosphate receptor; Genomic imprinting, Insulin-like growth factor; NMR {Gallus gallus}, error: None, warning: None
I0120 10:27:44.104951 140191300101952 af.py:224] I0120 18:27:44.104496 139969096478912 templates.py:718] hit 2m6t_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05442176870748299.
I0120 10:27:44.104995 140191300101952 af.py:224] I0120 18:27:44.104567 139969096478912 templates.py:913] Skipped invalid hit 2M6T_A Insulin-like growth factor 2 receptor; antitumor, directed evolution and high; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:44.105039 140191300101952 af.py:224] I0120 18:27:44.104615 139969096478912 templates.py:718] hit 2lla_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.054821928771508606.
I0120 10:27:44.105082 140191300101952 af.py:224] I0120 18:27:44.104696 139969096478912 templates.py:913] Skipped invalid hit 2LLA_A Mannose-6-phosphate/insulin-like growth factor II receptor; monotreme, mannose-6-phosphate, IGF-II, domain 11; NMR {Tachyglossus aculeatus}, error: None, warning: None
I0120 10:27:44.105126 140191300101952 af.py:224] I0120 18:27:44.104779 139969096478912 templates.py:718] hit 1gp0_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.054821928771508606.
I0120 10:27:44.105169 140191300101952 af.py:224] I0120 18:27:44.104816 139969096478912 templates.py:913] Skipped invalid hit 1GP0_A CATION-INDEPENDENT MANNOSE-6-PHOSPHATE RECEPTOR; RECEPTOR, INSULIN-LIKE GROWTH FACTOR, CATION; 1.4A {HOMO SAPIENS} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.105224 140191300101952 af.py:224] I0120 18:27:44.104858 139969096478912 templates.py:718] hit 2l21_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05442176870748299.
I0120 10:27:44.105269 140191300101952 af.py:224] I0120 18:27:44.104895 139969096478912 templates.py:913] Skipped invalid hit 2L21_A Cation-independent mannose-6-phosphate receptor; Genomic imprinting, Insulin-like growth factor; NMR {Gallus gallus}, error: None, warning: None
I0120 10:27:44.105313 140191300101952 af.py:224] I0120 18:27:44.104937 139969096478912 templates.py:718] hit 2l2g_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05442176870748299.
I0120 10:27:44.105357 140191300101952 af.py:224] I0120 18:27:44.104974 139969096478912 templates.py:913] Skipped invalid hit 2L2G_A entity; Insulin-like growth factor 2, mannose; NMR {Monodelphis domestica}, error: None, warning: None
I0120 10:27:44.108338 140191300101952 af.py:224] I0120 18:27:44.105017 139969096478912 templates.py:718] hit 2lla_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05322128851540616.
I0120 10:27:44.108403 140191300101952 af.py:224] I0120 18:27:44.105054 139969096478912 templates.py:913] Skipped invalid hit 2LLA_A Mannose-6-phosphate/insulin-like growth factor II receptor; monotreme, mannose-6-phosphate, IGF-II, domain 11; NMR {Tachyglossus aculeatus}, error: None, warning: None
I0120 10:27:44.108476 140191300101952 af.py:224] I0120 18:27:44.105098 139969096478912 templates.py:718] hit 1c39_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05442176870748299.
I0120 10:27:44.108525 140191300101952 af.py:224] I0120 18:27:44.105135 139969096478912 templates.py:913] Skipped invalid hit 1C39_A PROTEIN (CATION-DEPENDENT MANNOSE-6-PHOSPHATE RECEPTOR) MUTANT; RECEPTOR, CATION DEPENDENT MANNOSE 6-PHOSPHATE; HET: P3M, NAG; 1.85A {Bos taurus} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.108573 140191300101952 af.py:224] I0120 18:27:44.105180 139969096478912 templates.py:718] hit 2m6t_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05322128851540616.
I0120 10:27:44.108619 140191300101952 af.py:224] I0120 18:27:44.105217 139969096478912 templates.py:913] Skipped invalid hit 2M6T_A Insulin-like growth factor 2 receptor; antitumor, directed evolution and high; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:44.108680 140191300101952 af.py:224] I0120 18:27:44.105268 139969096478912 templates.py:718] hit 2rl8_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.055222088835534214.
I0120 10:27:44.108726 140191300101952 af.py:224] I0120 18:27:44.105308 139969096478912 templates.py:913] Skipped invalid hit 2RL8_A Cation-dependent mannose-6-phosphate receptor; P-type lectin, receptor, mannose 6-phosphate; HET: M6D, NAG; 1.45A {Bos taurus} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.108770 140191300101952 af.py:224] I0120 18:27:44.105350 139969096478912 templates.py:718] hit 2lla_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.053621448579431776.
I0120 10:27:44.108813 140191300101952 af.py:224] I0120 18:27:44.105387 139969096478912 templates.py:913] Skipped invalid hit 2LLA_A Mannose-6-phosphate/insulin-like growth factor II receptor; monotreme, mannose-6-phosphate, IGF-II, domain 11; NMR {Tachyglossus aculeatus}, error: None, warning: None
I0120 10:27:44.108857 140191300101952 af.py:224] I0120 18:27:44.105431 139969096478912 templates.py:718] hit 1gp0_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.053621448579431776.
I0120 10:27:44.108916 140191300101952 af.py:224] I0120 18:27:44.105468 139969096478912 templates.py:913] Skipped invalid hit 1GP0_A CATION-INDEPENDENT MANNOSE-6-PHOSPHATE RECEPTOR; RECEPTOR, INSULIN-LIKE GROWTH FACTOR, CATION; 1.4A {HOMO SAPIENS} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.108983 140191300101952 af.py:224] I0120 18:27:44.105513 139969096478912 templates.py:718] hit 2m6t_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05162064825930372.
I0120 10:27:44.109027 140191300101952 af.py:224] I0120 18:27:44.105550 139969096478912 templates.py:913] Skipped invalid hit 2M6T_A Insulin-like growth factor 2 receptor; antitumor, directed evolution and high; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:44.109071 140191300101952 af.py:224] I0120 18:27:44.105594 139969096478912 templates.py:718] hit 3k43_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.055222088835534214.
I0120 10:27:44.109115 140191300101952 af.py:224] I0120 18:27:44.105631 139969096478912 templates.py:913] Skipped invalid hit 3K43_A Cation-dependent mannose-6-phosphate receptor; transport, lysosome, mannose, receptor, sugar; HET: SO4, NAG; 2.0A {Bos taurus}, error: None, warning: None
I0120 10:27:44.109159 140191300101952 af.py:224] I0120 18:27:44.105679 139969096478912 templates.py:718] hit 2l2g_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05322128851540616.
I0120 10:27:44.109220 140191300101952 af.py:224] I0120 18:27:44.105718 139969096478912 templates.py:913] Skipped invalid hit 2L2G_A entity; Insulin-like growth factor 2, mannose; NMR {Monodelphis domestica}, error: None, warning: None
I0120 10:27:44.109263 140191300101952 af.py:224] I0120 18:27:44.105762 139969096478912 templates.py:718] hit 2rl8_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05402160864345738.
I0120 10:27:44.109306 140191300101952 af.py:224] I0120 18:27:44.105799 139969096478912 templates.py:913] Skipped invalid hit 2RL8_A Cation-dependent mannose-6-phosphate receptor; P-type lectin, receptor, mannose 6-phosphate; HET: M6D, NAG; 1.45A {Bos taurus} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.109349 140191300101952 af.py:224] I0120 18:27:44.105843 139969096478912 templates.py:718] hit 1gp0_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05202080832332933.
I0120 10:27:44.109392 140191300101952 af.py:224] I0120 18:27:44.105880 139969096478912 templates.py:913] Skipped invalid hit 1GP0_A CATION-INDEPENDENT MANNOSE-6-PHOSPHATE RECEPTOR; RECEPTOR, INSULIN-LIKE GROWTH FACTOR, CATION; 1.4A {HOMO SAPIENS} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.109435 140191300101952 af.py:224] I0120 18:27:44.105925 139969096478912 templates.py:718] hit 2m6t_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.052420968387354945.
I0120 10:27:44.109494 140191300101952 af.py:224] I0120 18:27:44.105962 139969096478912 templates.py:913] Skipped invalid hit 2M6T_A Insulin-like growth factor 2 receptor; antitumor, directed evolution and high; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:44.109538 140191300101952 af.py:224] I0120 18:27:44.106006 139969096478912 templates.py:718] hit 2rl8_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.053621448579431776.
I0120 10:27:44.109582 140191300101952 af.py:224] I0120 18:27:44.106043 139969096478912 templates.py:913] Skipped invalid hit 2RL8_A Cation-dependent mannose-6-phosphate receptor; P-type lectin, receptor, mannose 6-phosphate; HET: M6D, NAG; 1.45A {Bos taurus} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.109625 140191300101952 af.py:224] I0120 18:27:44.106093 139969096478912 templates.py:718] hit 1gp0_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05282112845138055.
I0120 10:27:44.109669 140191300101952 af.py:224] I0120 18:27:44.106132 139969096478912 templates.py:913] Skipped invalid hit 1GP0_A CATION-INDEPENDENT MANNOSE-6-PHOSPHATE RECEPTOR; RECEPTOR, INSULIN-LIKE GROWTH FACTOR, CATION; 1.4A {HOMO SAPIENS} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.109716 140191300101952 af.py:224] I0120 18:27:44.106176 139969096478912 templates.py:718] hit 2rl8_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.052420968387354945.
I0120 10:27:44.109760 140191300101952 af.py:224] I0120 18:27:44.106212 139969096478912 templates.py:913] Skipped invalid hit 2RL8_A Cation-dependent mannose-6-phosphate receptor; P-type lectin, receptor, mannose 6-phosphate; HET: M6D, NAG; 1.45A {Bos taurus} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.109821 140191300101952 af.py:224] I0120 18:27:44.106255 139969096478912 templates.py:718] hit 2lla_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05282112845138055.
I0120 10:27:44.109883 140191300101952 af.py:224] I0120 18:27:44.106292 139969096478912 templates.py:913] Skipped invalid hit 2LLA_A Mannose-6-phosphate/insulin-like growth factor II receptor; monotreme, mannose-6-phosphate, IGF-II, domain 11; NMR {Tachyglossus aculeatus}, error: None, warning: None
I0120 10:27:44.109927 140191300101952 af.py:224] I0120 18:27:44.106336 139969096478912 templates.py:718] hit 1c39_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.0508203281312525.
I0120 10:27:44.109971 140191300101952 af.py:224] I0120 18:27:44.106372 139969096478912 templates.py:913] Skipped invalid hit 1C39_A PROTEIN (CATION-DEPENDENT MANNOSE-6-PHOSPHATE RECEPTOR) MUTANT; RECEPTOR, CATION DEPENDENT MANNOSE 6-PHOSPHATE; HET: P3M, NAG; 1.85A {Bos taurus} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.110015 140191300101952 af.py:224] I0120 18:27:44.106416 139969096478912 templates.py:718] hit 1c39_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05162064825930372.
I0120 10:27:44.110076 140191300101952 af.py:224] I0120 18:27:44.106454 139969096478912 templates.py:913] Skipped invalid hit 1C39_A PROTEIN (CATION-DEPENDENT MANNOSE-6-PHOSPHATE RECEPTOR) MUTANT; RECEPTOR, CATION DEPENDENT MANNOSE 6-PHOSPHATE; HET: P3M, NAG; 1.85A {Bos taurus} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.110121 140191300101952 af.py:224] I0120 18:27:44.106498 139969096478912 templates.py:718] hit 3k43_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05162064825930372.
I0120 10:27:44.110164 140191300101952 af.py:224] I0120 18:27:44.106536 139969096478912 templates.py:913] Skipped invalid hit 3K43_A Cation-dependent mannose-6-phosphate receptor; transport, lysosome, mannose, receptor, sugar; HET: SO4, NAG; 2.0A {Bos taurus}, error: None, warning: None
I0120 10:27:44.110207 140191300101952 af.py:224] I0120 18:27:44.106579 139969096478912 templates.py:718] hit 1c39_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05042016806722689.
I0120 10:27:44.110250 140191300101952 af.py:224] I0120 18:27:44.106616 139969096478912 templates.py:913] Skipped invalid hit 1C39_A PROTEIN (CATION-DEPENDENT MANNOSE-6-PHOSPHATE RECEPTOR) MUTANT; RECEPTOR, CATION DEPENDENT MANNOSE 6-PHOSPHATE; HET: P3M, NAG; 1.85A {Bos taurus} SCOP: b.64.1.1, error: None, warning: None
I0120 10:27:44.110294 140191300101952 af.py:224] I0120 18:27:44.106665 139969096478912 templates.py:718] hit 3k43_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.05042016806722689.
I0120 10:27:44.110337 140191300101952 af.py:224] I0120 18:27:44.106705 139969096478912 templates.py:913] Skipped invalid hit 3K43_A Cation-dependent mannose-6-phosphate receptor; transport, lysosome, mannose, receptor, sugar; HET: SO4, NAG; 2.0A {Bos taurus}, error: None, warning: None
I0120 10:27:44.110396 140191300101952 af.py:224] I0120 18:27:44.106748 139969096478912 templates.py:718] hit 3k43_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.04881952781112445.
I0120 10:27:44.110440 140191300101952 af.py:224] I0120 18:27:44.106785 139969096478912 templates.py:913] Skipped invalid hit 3K43_A Cation-dependent mannose-6-phosphate receptor; transport, lysosome, mannose, receptor, sugar; HET: SO4, NAG; 2.0A {Bos taurus}, error: None, warning: None
I0120 10:27:44.110489 140191300101952 af.py:224] I0120 18:27:44.106828 139969096478912 templates.py:718] hit 2n1h_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.037615046018407365.
I0120 10:27:44.110533 140191300101952 af.py:224] I0120 18:27:44.106865 139969096478912 templates.py:913] Skipped invalid hit 2N1H_A Glucosidase 2 subunit beta; hydrolase, lectin, MRH; NMR {Schizosaccharomyces pombe}, error: None, warning: None
I0120 10:27:44.110576 140191300101952 af.py:224] I0120 18:27:44.106909 139969096478912 templates.py:718] hit 4xqm_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.037615046018407365.
I0120 10:27:44.110619 140191300101952 af.py:224] I0120 18:27:44.106947 139969096478912 templates.py:913] Skipped invalid hit 4XQM_A MRH domain of Glucosidase II; hydrolase, sugar binding protein; HET: MAN; 1.625A {Schizosaccharomyces pombe (strain 972 / ATCC 24843)}, error: None, warning: None
I0120 10:27:44.110663 140191300101952 af.py:224] I0120 18:27:44.106991 139969096478912 templates.py:718] hit 4xqm_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.03721488595438175.
I0120 10:27:44.110706 140191300101952 af.py:224] I0120 18:27:44.107028 139969096478912 templates.py:913] Skipped invalid hit 4XQM_A MRH domain of Glucosidase II; hydrolase, sugar binding protein; HET: MAN; 1.625A {Schizosaccharomyces pombe (strain 972 / ATCC 24843)}, error: None, warning: None
I0120 10:27:44.110749 140191300101952 af.py:224] I0120 18:27:44.107076 139969096478912 templates.py:718] hit 2n1h_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.03721488595438175.
I0120 10:27:44.110793 140191300101952 af.py:224] I0120 18:27:44.107115 139969096478912 templates.py:913] Skipped invalid hit 2N1H_A Glucosidase 2 subunit beta; hydrolase, lectin, MRH; NMR {Schizosaccharomyces pombe}, error: None, warning: None
I0120 10:27:44.110836 140191300101952 af.py:224] I0120 18:27:44.107161 139969096478912 templates.py:718] hit 4xqm_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.03201280512204882.
I0120 10:27:44.110879 140191300101952 af.py:224] I0120 18:27:44.107199 139969096478912 templates.py:913] Skipped invalid hit 4XQM_A MRH domain of Glucosidase II; hydrolase, sugar binding protein; HET: MAN; 1.625A {Schizosaccharomyces pombe (strain 972 / ATCC 24843)}, error: None, warning: None
I0120 10:27:44.110922 140191300101952 af.py:224] I0120 18:27:44.107241 139969096478912 templates.py:718] hit 2n1h_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.03201280512204882.
I0120 10:27:44.110965 140191300101952 af.py:224] I0120 18:27:44.107278 139969096478912 templates.py:913] Skipped invalid hit 2N1H_A Glucosidase 2 subunit beta; hydrolase, lectin, MRH; NMR {Schizosaccharomyces pombe}, error: None, warning: None
I0120 10:27:44.111009 140191300101952 af.py:224] I0120 18:27:44.107321 139969096478912 templates.py:718] hit 4xqm_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.03201280512204882.
I0120 10:27:44.111052 140191300101952 af.py:224] I0120 18:27:44.107358 139969096478912 templates.py:913] Skipped invalid hit 4XQM_A MRH domain of Glucosidase II; hydrolase, sugar binding protein; HET: MAN; 1.625A {Schizosaccharomyces pombe (strain 972 / ATCC 24843)}, error: None, warning: None
I0120 10:27:44.111094 140191300101952 af.py:224] I0120 18:27:44.107401 139969096478912 templates.py:718] hit 2n1h_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.025610244097639057.
I0120 10:27:44.111137 140191300101952 af.py:224] I0120 18:27:44.107437 139969096478912 templates.py:913] Skipped invalid hit 2N1H_A Glucosidase 2 subunit beta; hydrolase, lectin, MRH; NMR {Schizosaccharomyces pombe}, error: None, warning: None
I0120 10:27:44.111184 140191300101952 af.py:224] I0120 18:27:44.107479 139969096478912 templates.py:718] hit 6f9a_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.025210084033613446.
I0120 10:27:44.111228 140191300101952 af.py:224] I0120 18:27:44.107516 139969096478912 templates.py:913] Skipped invalid hit 6F9A_A ER quality-control lectin; MRH domain, beta barrel, lectin; NMR {Saccharomyces cerevisiae}, error: None, warning: None
I0120 10:27:44.111271 140191300101952 af.py:224] I0120 18:27:44.107559 139969096478912 templates.py:718] hit 6f9a_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.023209283713485393.
I0120 10:27:44.111314 140191300101952 af.py:224] I0120 18:27:44.107596 139969096478912 templates.py:913] Skipped invalid hit 6F9A_A ER quality-control lectin; MRH domain, beta barrel, lectin; NMR {Saccharomyces cerevisiae}, error: None, warning: None
I0120 10:27:44.111358 140191300101952 af.py:224] I0120 18:27:44.107638 139969096478912 templates.py:718] hit 6f9a_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.024809923969587835.
I0120 10:27:44.111401 140191300101952 af.py:224] I0120 18:27:44.107673 139969096478912 templates.py:913] Skipped invalid hit 6F9A_A ER quality-control lectin; MRH domain, beta barrel, lectin; NMR {Saccharomyces cerevisiae}, error: None, warning: None
I0120 10:27:44.111444 140191300101952 af.py:224] I0120 18:27:44.107713 139969096478912 templates.py:718] hit 6f9a_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.022809123649459785.
I0120 10:27:44.111487 140191300101952 af.py:224] I0120 18:27:44.107750 139969096478912 templates.py:913] Skipped invalid hit 6F9A_A ER quality-control lectin; MRH domain, beta barrel, lectin; NMR {Saccharomyces cerevisiae}, error: None, warning: None
I0120 10:27:44.111545 140191300101952 af.py:224] I0120 18:27:44.107798 139969096478912 templates.py:718] hit 3aih_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.022809123649459785.
I0120 10:27:44.111591 140191300101952 af.py:224] I0120 18:27:44.107837 139969096478912 templates.py:913] Skipped invalid hit 3AIH_B Protein OS-9; Beta barrel, Lectin, SUGAR BINDING; HET: BMA, MAN; 2.1A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.111634 140191300101952 af.py:224] I0120 18:27:44.107880 139969096478912 templates.py:718] hit 3aih_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.024409763905562223.
I0120 10:27:44.111677 140191300101952 af.py:224] I0120 18:27:44.107916 139969096478912 templates.py:913] Skipped invalid hit 3AIH_B Protein OS-9; Beta barrel, Lectin, SUGAR BINDING; HET: BMA, MAN; 2.1A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.111720 140191300101952 af.py:224] I0120 18:27:44.107958 139969096478912 templates.py:718] hit 1eak_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01920768307322929.
I0120 10:27:44.111763 140191300101952 af.py:224] I0120 18:27:44.107993 139969096478912 templates.py:913] Skipped invalid hit 1EAK_A 72 KDA TYPE IV COLLAGENASE; HYDROLASE-HYDROLASE INHIBITOR COMPLEX, HYDROLYSE, MATRIX; HET: SO4; 2.66A {HOMO SAPIENS} SCOP: d.92.1.11, a.20.1.2, g.14.1.2, error: None, warning: None
I0120 10:27:44.111808 140191300101952 af.py:224] I0120 18:27:44.108034 139969096478912 templates.py:718] hit 1eak_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01920768307322929.
I0120 10:27:44.111851 140191300101952 af.py:224] I0120 18:27:44.108070 139969096478912 templates.py:913] Skipped invalid hit 1EAK_B 72 KDA TYPE IV COLLAGENASE; HYDROLASE-HYDROLASE INHIBITOR COMPLEX, HYDROLYSE, MATRIX; HET: SO4; 2.66A {HOMO SAPIENS} SCOP: a.20.1.2, d.92.1.11, g.14.1.2, error: None, warning: None
I0120 10:27:44.111894 140191300101952 af.py:224] I0120 18:27:44.108112 139969096478912 templates.py:718] hit 3aih_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.023609443777511004.
I0120 10:27:44.111942 140191300101952 af.py:224] I0120 18:27:44.108156 139969096478912 templates.py:913] Skipped invalid hit 3AIH_B Protein OS-9; Beta barrel, Lectin, SUGAR BINDING; HET: BMA, MAN; 2.1A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.111985 140191300101952 af.py:224] I0120 18:27:44.108198 139969096478912 templates.py:718] hit 3aih_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.023609443777511004.
I0120 10:27:44.112028 140191300101952 af.py:224] I0120 18:27:44.108235 139969096478912 templates.py:913] Skipped invalid hit 3AIH_B Protein OS-9; Beta barrel, Lectin, SUGAR BINDING; HET: BMA, MAN; 2.1A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.112071 140191300101952 af.py:224] I0120 18:27:44.108277 139969096478912 templates.py:718] hit 1eak_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112119 140191300101952 af.py:224] I0120 18:27:44.108313 139969096478912 templates.py:913] Skipped invalid hit 1EAK_B 72 KDA TYPE IV COLLAGENASE; HYDROLASE-HYDROLASE INHIBITOR COMPLEX, HYDROLYSE, MATRIX; HET: SO4; 2.66A {HOMO SAPIENS} SCOP: a.20.1.2, d.92.1.11, g.14.1.2, error: None, warning: None
I0120 10:27:44.112164 140191300101952 af.py:224] I0120 18:27:44.108355 139969096478912 templates.py:718] hit 1eak_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112207 140191300101952 af.py:224] I0120 18:27:44.108392 139969096478912 templates.py:913] Skipped invalid hit 1EAK_A 72 KDA TYPE IV COLLAGENASE; HYDROLASE-HYDROLASE INHIBITOR COMPLEX, HYDROLYSE, MATRIX; HET: SO4; 2.66A {HOMO SAPIENS} SCOP: d.92.1.11, a.20.1.2, g.14.1.2, error: None, warning: None
I0120 10:27:44.112251 140191300101952 af.py:224] I0120 18:27:44.108434 139969096478912 templates.py:718] hit 1l6j_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112294 140191300101952 af.py:224] I0120 18:27:44.108471 139969096478912 templates.py:913] Skipped invalid hit 1L6J_A Matrix metalloproteinase-9 (E.C. 3.4.24.35); Twisted beta sheet flanked by; 2.5A {Homo sapiens} SCOP: g.14.1.2, a.20.1.2, d.92.1.11, error: None, warning: None
I0120 10:27:44.112337 140191300101952 af.py:224] I0120 18:27:44.108513 139969096478912 templates.py:718] hit 1eak_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112380 140191300101952 af.py:224] I0120 18:27:44.108548 139969096478912 templates.py:913] Skipped invalid hit 1EAK_A 72 KDA TYPE IV COLLAGENASE; HYDROLASE-HYDROLASE INHIBITOR COMPLEX, HYDROLYSE, MATRIX; HET: SO4; 2.66A {HOMO SAPIENS} SCOP: d.92.1.11, a.20.1.2, g.14.1.2, error: None, warning: None
I0120 10:27:44.112423 140191300101952 af.py:224] I0120 18:27:44.108589 139969096478912 templates.py:718] hit 1eak_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112467 140191300101952 af.py:224] I0120 18:27:44.108627 139969096478912 templates.py:913] Skipped invalid hit 1EAK_B 72 KDA TYPE IV COLLAGENASE; HYDROLASE-HYDROLASE INHIBITOR COMPLEX, HYDROLYSE, MATRIX; HET: SO4; 2.66A {HOMO SAPIENS} SCOP: a.20.1.2, d.92.1.11, g.14.1.2, error: None, warning: None
I0120 10:27:44.112510 140191300101952 af.py:224] I0120 18:27:44.108674 139969096478912 templates.py:718] hit 1l6j_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01920768307322929.
I0120 10:27:44.112553 140191300101952 af.py:224] I0120 18:27:44.108713 139969096478912 templates.py:913] Skipped invalid hit 1L6J_A Matrix metalloproteinase-9 (E.C. 3.4.24.35); Twisted beta sheet flanked by; 2.5A {Homo sapiens} SCOP: g.14.1.2, a.20.1.2, d.92.1.11, error: None, warning: None
I0120 10:27:44.112596 140191300101952 af.py:224] I0120 18:27:44.108755 139969096478912 templates.py:718] hit 1ck7_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112644 140191300101952 af.py:224] I0120 18:27:44.108791 139969096478912 templates.py:913] Skipped invalid hit 1CK7_A GELATINASE A; HYDROLASE (METALLOPROTEASE), FULL-LENGTH, METALLOPROTEINASE, GELATINASE; HET: SO4; 2.8A {Homo sapiens} SCOP: b.66.1.1, d.92.1.11, a.20.1.2, g.14.1.2, error: None, warning: None
I0120 10:27:44.112688 140191300101952 af.py:224] I0120 18:27:44.108833 139969096478912 templates.py:718] hit 1gxd_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112731 140191300101952 af.py:224] I0120 18:27:44.108871 139969096478912 templates.py:913] Skipped invalid hit 1GXD_A 72 KDA TYPE IV COLLAGENASE; HYDROLASE, METALLOPROTEASE, ZYMOGEN, COLLAGEN DEGRADATION; HET: SO4; 3.1A {HOMO SAPIENS} SCOP: b.66.1.1, d.92.1.11, a.20.1.2, g.14.1.2, error: None, warning: None
I0120 10:27:44.112774 140191300101952 af.py:224] I0120 18:27:44.108913 139969096478912 templates.py:718] hit 3m7p_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112818 140191300101952 af.py:224] I0120 18:27:44.108949 139969096478912 templates.py:913] Skipped invalid hit 3M7P_A FN1 protein; fibronectin, extracellular matrix, modular protein; HET: 12P, NAG; 2.5A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.112861 140191300101952 af.py:224] I0120 18:27:44.108990 139969096478912 templates.py:718] hit 5ew6_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112904 140191300101952 af.py:224] I0120 18:27:44.109027 139969096478912 templates.py:913] Skipped invalid hit 5EW6_A C-type mannose receptor 2; endocytic collagen receptor, SUGAR BINDING; HET: 1PE, NAG; 2.29A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.112947 140191300101952 af.py:224] I0120 18:27:44.109068 139969096478912 templates.py:718] hit 1ck7_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.112991 140191300101952 af.py:224] I0120 18:27:44.109105 139969096478912 templates.py:913] Skipped invalid hit 1CK7_A GELATINASE A; HYDROLASE (METALLOPROTEASE), FULL-LENGTH, METALLOPROTEINASE, GELATINASE; HET: SO4; 2.8A {Homo sapiens} SCOP: b.66.1.1, d.92.1.11, a.20.1.2, g.14.1.2, error: None, warning: None
I0120 10:27:44.113035 140191300101952 af.py:224] I0120 18:27:44.109146 139969096478912 templates.py:718] hit 1gxd_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.113079 140191300101952 af.py:224] I0120 18:27:44.109181 139969096478912 templates.py:913] Skipped invalid hit 1GXD_A 72 KDA TYPE IV COLLAGENASE; HYDROLASE, METALLOPROTEASE, ZYMOGEN, COLLAGEN DEGRADATION; HET: SO4; 3.1A {HOMO SAPIENS} SCOP: b.66.1.1, d.92.1.11, a.20.1.2, g.14.1.2, error: None, warning: None
I0120 10:27:44.113122 140191300101952 af.py:224] I0120 18:27:44.109223 139969096478912 templates.py:718] hit 3mql_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.113188 140191300101952 af.py:224] I0120 18:27:44.109260 139969096478912 templates.py:913] Skipped invalid hit 3MQL_A Fibronectin 1; CELL ADHESION, fibronectin, collagen, GBD; HET: NAG, MRD; 3.004A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.113233 140191300101952 af.py:224] I0120 18:27:44.109302 139969096478912 templates.py:718] hit 5xts_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.113277 140191300101952 af.py:224] I0120 18:27:44.109338 139969096478912 templates.py:913] Skipped invalid hit 5XTS_A Macrophage mannose receptor 1; Collagen binding, Lectin-activity, Endocytic receptor; 2.0A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.113325 140191300101952 af.py:224] I0120 18:27:44.109380 139969096478912 templates.py:718] hit 6inn_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.113369 140191300101952 af.py:224] I0120 18:27:44.109415 139969096478912 templates.py:913] Skipped invalid hit 6INN_A Macrophage mannose receptor 1; CD206, mannose receptor family, pH-dependent; HET: MSE; 3.001A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.113413 140191300101952 af.py:224] I0120 18:27:44.109457 139969096478912 templates.py:718] hit 1e88_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.113456 140191300101952 af.py:224] I0120 18:27:44.109494 139969096478912 templates.py:913] Skipped invalid hit 1E88_A FIBRONECTIN; EXTRACELLULAR MATRIX GLYCOPROTEIN, CELL ADHESION; HET: NAG; NMR {HOMO SAPIENS} SCOP: g.27.1.1, g.14.1.2, error: None, warning: None
I0120 10:27:44.113500 140191300101952 af.py:224] I0120 18:27:44.109542 139969096478912 templates.py:718] hit 1j7m_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.113544 140191300101952 af.py:224] I0120 18:27:44.109581 139969096478912 templates.py:913] Skipped invalid hit 1J7M_A 72 KDA TYPE IV COLLAGENASE; beta sheet, alpha helix, HYDROLASE; NMR {Homo sapiens} SCOP: g.14.1.2, error: None, warning: None
I0120 10:27:44.113587 140191300101952 af.py:224] I0120 18:27:44.109622 139969096478912 templates.py:718] hit 1qo6_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.113630 140191300101952 af.py:224] I0120 18:27:44.109658 139969096478912 templates.py:913] Skipped invalid hit 1QO6_A FIBRONECTIN; CELL ADHESION PROTEIN, FIBRONECTIN MODULE; NMR {HOMO SAPIENS} SCOP: g.14.1.2, g.27.1.1, error: None, warning: None
I0120 10:27:44.113672 140191300101952 af.py:224] I0120 18:27:44.109699 139969096478912 templates.py:718] hit 3m7p_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.113716 140191300101952 af.py:224] I0120 18:27:44.109735 139969096478912 templates.py:913] Skipped invalid hit 3M7P_A FN1 protein; fibronectin, extracellular matrix, modular protein; HET: 12P, NAG; 2.5A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.113759 140191300101952 af.py:224] I0120 18:27:44.109776 139969096478912 templates.py:718] hit 5ao5_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.113802 140191300101952 af.py:224] I0120 18:27:44.109813 139969096478912 templates.py:913] Skipped invalid hit 5AO5_A C-TYPE MANNOSE RECEPTOR 2; ENDOCYTOSIS, ENDOCYTIC RECEPTOR, FIBRONECTIN TYPE; HET: SO4; 2.48A {HOMO SAPIENS}, error: None, warning: None
I0120 10:27:44.113845 140191300101952 af.py:224] I0120 18:27:44.109854 139969096478912 templates.py:718] hit 1e88_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.113888 140191300101952 af.py:224] I0120 18:27:44.109890 139969096478912 templates.py:913] Skipped invalid hit 1E88_A FIBRONECTIN; EXTRACELLULAR MATRIX GLYCOPROTEIN, CELL ADHESION; HET: NAG; NMR {HOMO SAPIENS} SCOP: g.27.1.1, g.14.1.2, error: None, warning: None
I0120 10:27:44.113932 140191300101952 af.py:224] I0120 18:27:44.109932 139969096478912 templates.py:718] hit 3mql_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.113976 140191300101952 af.py:224] I0120 18:27:44.109967 139969096478912 templates.py:913] Skipped invalid hit 3MQL_A Fibronectin 1; CELL ADHESION, fibronectin, collagen, GBD; HET: NAG, MRD; 3.004A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.114020 140191300101952 af.py:224] I0120 18:27:44.110009 139969096478912 templates.py:718] hit 1ck7_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.114067 140191300101952 af.py:224] I0120 18:27:44.110045 139969096478912 templates.py:913] Skipped invalid hit 1CK7_A GELATINASE A; HYDROLASE (METALLOPROTEASE), FULL-LENGTH, METALLOPROTEINASE, GELATINASE; HET: SO4; 2.8A {Homo sapiens} SCOP: b.66.1.1, d.92.1.11, a.20.1.2, g.14.1.2, error: None, warning: None
I0120 10:27:44.114111 140191300101952 af.py:224] I0120 18:27:44.110086 139969096478912 templates.py:718] hit 1gxd_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.018807523009203683.
I0120 10:27:44.114155 140191300101952 af.py:224] I0120 18:27:44.110123 139969096478912 templates.py:913] Skipped invalid hit 1GXD_A 72 KDA TYPE IV COLLAGENASE; HYDROLASE, METALLOPROTEASE, ZYMOGEN, COLLAGEN DEGRADATION; HET: SO4; 3.1A {HOMO SAPIENS} SCOP: b.66.1.1, d.92.1.11, a.20.1.2, g.14.1.2, error: None, warning: None
I0120 10:27:44.114199 140191300101952 af.py:224] I0120 18:27:44.110165 139969096478912 templates.py:718] hit 1ks0_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.114242 140191300101952 af.py:224] I0120 18:27:44.110202 139969096478912 templates.py:913] Skipped invalid hit 1KS0_A Matrix Metalloproteinase 2 (E.C.3.4.24.24); beta sheet, alpha helix, 3/1; NMR {Homo sapiens} SCOP: g.14.1.2, error: None, warning: None
I0120 10:27:44.114286 140191300101952 af.py:224] I0120 18:27:44.110244 139969096478912 templates.py:718] hit 1l6j_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.114330 140191300101952 af.py:224] I0120 18:27:44.110279 139969096478912 templates.py:913] Skipped invalid hit 1L6J_A Matrix metalloproteinase-9 (E.C. 3.4.24.35); Twisted beta sheet flanked by; 2.5A {Homo sapiens} SCOP: g.14.1.2, a.20.1.2, d.92.1.11, error: None, warning: None
I0120 10:27:44.114374 140191300101952 af.py:224] I0120 18:27:44.110320 139969096478912 templates.py:718] hit 2fn2_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.114418 140191300101952 af.py:224] I0120 18:27:44.110357 139969096478912 templates.py:913] Skipped invalid hit 2FN2_A FIBRONECTIN; GLYCOPROTEIN, FIBRONECTIN, TYPE TWO MODULE; HET: NAG; NMR {Homo sapiens} SCOP: g.14.1.2, error: None, warning: None
I0120 10:27:44.114461 140191300101952 af.py:224] I0120 18:27:44.110405 139969096478912 templates.py:718] hit 1cxw_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.114505 140191300101952 af.py:224] I0120 18:27:44.110445 139969096478912 templates.py:913] Skipped invalid hit 1CXW_A HUMAN MATRIX METALLOPROTEINASE 2 (E.C.3.4.24.24); BETA SHEET, ALPHA HELIX, HYDROLASE; NMR {Homo sapiens} SCOP: g.14.1.2, error: None, warning: None
I0120 10:27:44.114548 140191300101952 af.py:224] I0120 18:27:44.110486 139969096478912 templates.py:718] hit 1h8p_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01800720288115246.
I0120 10:27:44.114592 140191300101952 af.py:224] I0120 18:27:44.110522 139969096478912 templates.py:913] Skipped invalid hit 1H8P_B SEMINAL PLASMA PROTEIN PDC-109; PHOSPHORYLCHOLINE-BINDING PROTEIN; HET: PC; 1.82A {BOS TAURUS} SCOP: g.14.1.2, error: None, warning: None
I0120 10:27:44.114636 140191300101952 af.py:224] I0120 18:27:44.110564 139969096478912 templates.py:718] hit 6inu_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.114679 140191300101952 af.py:224] I0120 18:27:44.110601 139969096478912 templates.py:913] Skipped invalid hit 6INU_B Macrophage mannose receptor 1; CD206, mannose receptor family, pH-dependent; 2.65A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.114722 140191300101952 af.py:224] I0120 18:27:44.110642 139969096478912 templates.py:718] hit 6inu_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.114769 140191300101952 af.py:224] I0120 18:27:44.110679 139969096478912 templates.py:913] Skipped invalid hit 6INU_A Macrophage mannose receptor 1; CD206, mannose receptor family, pH-dependent; 2.65A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.114814 140191300101952 af.py:224] I0120 18:27:44.110720 139969096478912 templates.py:718] hit 6ino_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01840736294517807.
I0120 10:27:44.114877 140191300101952 af.py:224] I0120 18:27:44.110755 139969096478912 templates.py:913] Skipped invalid hit 6INO_A Macrophage mannose receptor 1; CD206, mannose receptor family, pH-dependent; 3.054A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.114922 140191300101952 af.py:224] I0120 18:27:44.110797 139969096478912 templates.py:718] hit 1pdc_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01720688275310124.
I0120 10:27:44.114965 140191300101952 af.py:224] I0120 18:27:44.110833 139969096478912 templates.py:913] Skipped invalid hit 1PDC_A SEMINAL FLUID PROTEIN PDC-109 (DOMAIN; COLLAGEN-BINDING TYPE II DOMAIN; NMR {Bos taurus} SCOP: g.14.1.2, error: None, warning: None
I0120 10:27:44.115009 140191300101952 af.py:224] I0120 18:27:44.110875 139969096478912 templates.py:718] hit 1h8p_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01800720288115246.
I0120 10:27:44.115053 140191300101952 af.py:224] I0120 18:27:44.110912 139969096478912 templates.py:913] Skipped invalid hit 1H8P_B SEMINAL PLASMA PROTEIN PDC-109; PHOSPHORYLCHOLINE-BINDING PROTEIN; HET: PC; 1.82A {BOS TAURUS} SCOP: g.14.1.2, error: None, warning: None
I0120 10:27:44.115096 140191300101952 af.py:224] I0120 18:27:44.110953 139969096478912 templates.py:718] hit 3mql_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.01640656262505002.
I0120 10:27:44.115140 140191300101952 af.py:224] I0120 18:27:44.110989 139969096478912 templates.py:913] Skipped invalid hit 3MQL_A Fibronectin 1; CELL ADHESION, fibronectin, collagen, GBD; HET: NAG, MRD; 3.004A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.115184 140191300101952 af.py:224] I0120 18:27:44.111031 139969096478912 templates.py:718] hit 6n5x_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.012404961984793917.
I0120 10:27:44.115228 140191300101952 af.py:224] I0120 18:27:44.111068 139969096478912 templates.py:913] Skipped invalid hit 6N5X_A Sorting nexin-5,Cation-independent mannose-6-phosphate receptor; sorting nexin, SNX, endosome, ENDOCYTOSIS; HET: JEF, GOL; 2.051A {Homo sapiens}, error: None, warning: None
I0120 10:27:44.115272 140191300101952 af.py:224] I0120 18:27:44.111115 139969096478912 templates.py:718] hit 2n28_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.004801920768307323.
I0120 10:27:44.115315 140191300101952 af.py:224] I0120 18:27:44.111154 139969096478912 templates.py:913] Skipped invalid hit 2N28_A Protein Vpu; alpha helix, VIRAL PROTEIN; NMR {Human immunodeficiency virus 1}, error: None, warning: None
I0120 10:27:44.115359 140191300101952 af.py:224] I0120 18:27:44.111197 139969096478912 templates.py:718] hit 2klu_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.004001600640256103.
I0120 10:27:44.115402 140191300101952 af.py:224] I0120 18:27:44.111233 139969096478912 templates.py:913] Skipped invalid hit 2KLU_A T-cell surface glycoprotein CD4; Protein, Cell membrane, Disulfide bond; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:44.115445 140191300101952 af.py:224] I0120 18:27:44.111276 139969096478912 templates.py:718] hit 5lv6_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.003601440576230492.
I0120 10:27:44.115489 140191300101952 af.py:224] I0120 18:27:44.111312 139969096478912 templates.py:913] Skipped invalid hit 5LV6_B Epidermal growth factor receptor (E.C.2.7.10.1); Epidermal growth factor receptor, bicelles; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:44.115544 140191300101952 af.py:224] I0120 18:27:44.111354 139969096478912 templates.py:718] hit 2k9y_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.003201280512204882.
I0120 10:27:44.115589 140191300101952 af.py:224] I0120 18:27:44.111391 139969096478912 templates.py:913] Skipped invalid hit 2K9Y_B Ephrin type-A receptor 2 (E.C.2.7.10.1); receptor tyrosine kinase, membrane protein; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:44.115632 140191300101952 af.py:224] I0120 18:27:44.111433 139969096478912 templates.py:718] hit 2knc_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.0024009603841536613.
I0120 10:27:44.115676 140191300101952 af.py:224] I0120 18:27:44.111470 139969096478912 templates.py:913] Skipped invalid hit 2KNC_B Integrin alpha-IIb, Integrin beta-3; integrin, transmembrane signaling, protein structure; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:44.115719 140191300101952 af.py:224] I0120 18:27:44.111513 139969096478912 templates.py:718] hit 5zaz_A did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.0024009603841536613.
I0120 10:27:44.115763 140191300101952 af.py:224] I0120 18:27:44.111550 139969096478912 templates.py:913] Skipped invalid hit 5ZAZ_A Integrin beta-2; integrin b2, protein-lipid interaction, phospholipids; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:44.115806 140191300101952 af.py:224] I0120 18:27:44.111597 139969096478912 templates.py:718] hit 2lcx_B did not pass prefilter: Proportion of residues aligned to query too small. Align ratio: 0.0024009603841536613.
I0120 10:27:44.115849 140191300101952 af.py:224] I0120 18:27:44.111636 139969096478912 templates.py:913] Skipped invalid hit 2LCX_B Receptor tyrosine-protein kinase erbB-4 (E.C.2.7.10.1); tranmsembrane, helical dimer, dimerization, HER; NMR {Homo sapiens}, error: None, warning: None
I0120 10:27:50.349354 140191300101952 af.py:224] I0120 18:27:50.348843 139969096478912 pipeline.py:221] Uniref90 MSA size: 5368 sequences.
I0120 10:27:50.349463 140191300101952 af.py:224] I0120 18:27:50.348966 139969096478912 pipeline.py:222] BFD MSA size: 11899 sequences.
I0120 10:27:50.349516 140191300101952 af.py:224] I0120 18:27:50.349012 139969096478912 pipeline.py:223] MGnify MSA size: 217 sequences.
I0120 10:27:50.349575 140191300101952 af.py:224] I0120 18:27:50.349050 139969096478912 pipeline.py:225] Final (deduplicated) MSA size: 16908 sequences.
I0120 10:27:50.349622 140191300101952 af.py:224] I0120 18:27:50.349201 139969096478912 pipeline.py:228] Total number of templates (NB: this can include bad templates and is later filtered to top 4): 19.
I0120 10:27:50.631486 140191300101952 af.py:224] I0120 18:27:50.631031 139969096478912 run_alphafold.py:185] Running model model_1 on seq_6um1
I0120 10:28:00.957210 140191300101952 af.py:224] 2022-01-20 18:28:00.956198: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 3718272096 exceeds 10% of free system memory.
I0120 10:28:00.978644 140191300101952 af.py:224] 2022-01-20 18:28:00.978015: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 3770491200 exceeds 10% of free system memory.
I0120 10:28:03.664853 140191300101952 af.py:224] 2022-01-20 18:28:03.663906: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 3770491200 exceeds 10% of free system memory.
I0120 10:28:05.561131 140191300101952 af.py:224] 2022-01-20 18:28:05.559822: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 3770491200 exceeds 10% of free system memory.
I0120 10:28:07.468869 140191300101952 af.py:224] 2022-01-20 18:28:07.467254: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 3770491200 exceeds 10% of free system memory.
I0120 10:28:09.491443 140191300101952 af.py:224] I0120 18:28:09.490948 139969096478912 model.py:166] Running predict with shape(feat) = {'aatype': (4, 2499), 'residue_index': (4, 2499), 'seq_length': (4,), 'template_aatype': (4, 4, 2499), 'template_all_atom_masks': (4, 4, 2499, 37), 'template_all_atom_positions': (4, 4, 2499, 37, 3), 'template_sum_probs': (4, 4, 1), 'is_distillation': (4,), 'seq_mask': (4, 2499), 'msa_mask': (4, 508, 2499), 'msa_row_mask': (4, 508), 'random_crop_to_size_seed': (4, 2), 'template_mask': (4, 4), 'template_pseudo_beta': (4, 4, 2499, 3), 'template_pseudo_beta_mask': (4, 4, 2499), 'atom14_atom_exists': (4, 2499, 14), 'residx_atom14_to_atom37': (4, 2499, 14), 'residx_atom37_to_atom14': (4, 2499, 37), 'atom37_atom_exists': (4, 2499, 37), 'extra_msa': (4, 5120, 2499), 'extra_msa_mask': (4, 5120, 2499), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 508, 2499), 'true_msa': (4, 508, 2499), 'extra_has_deletion': (4, 5120, 2499), 'extra_deletion_value': (4, 5120, 2499), 'msa_feat': (4, 508, 2499, 49), 'target_feat': (4, 2499, 22)}
I0120 10:33:54.298780 140191300101952 af.py:224] 2022-01-20 18:33:54.297314: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:2086] Execution of replica 0 failed: INTERNAL: Failed to launch CUDA kernel: fusion_189 with block dimensions: 256x1x1 and grid dimensions: 625x1x1: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:54.308091 140191300101952 af.py:224] Traceback (most recent call last):
I0120 10:33:54.308192 140191300101952 af.py:224] File "/app/alphafold/run_alphafold.py", line 427, in <module>
I0120 10:33:54.308247 140191300101952 af.py:224] app.run(main)
I0120 10:33:54.308292 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 312, in run
I0120 10:33:54.308337 140191300101952 af.py:224] _run_main(main, args)
I0120 10:33:54.308379 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 258, in _run_main
I0120 10:33:54.308424 140191300101952 af.py:224] sys.exit(main(argv))
I0120 10:33:54.308466 140191300101952 af.py:224] File "/app/alphafold/run_alphafold.py", line 412, in main
I0120 10:33:54.308508 140191300101952 af.py:224] is_prokaryote=is_prokaryote)
I0120 10:33:54.308573 140191300101952 af.py:224] File "/app/alphafold/run_alphafold.py", line 194, in predict_structure
I0120 10:33:54.308616 140191300101952 af.py:224] random_seed=model_random_seed)
I0120 10:33:54.308658 140191300101952 af.py:224] File "/app/alphafold/alphafold/model/model.py", line 167, in predict
I0120 10:33:54.308700 140191300101952 af.py:224] result = self.apply(self.params, jax.random.PRNGKey(random_seed), feat)
I0120 10:33:54.308741 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
I0120 10:33:54.308782 140191300101952 af.py:224] return fun(*args, **kwargs)
I0120 10:33:54.308824 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/jax/_src/api.py", line 427, in cache_miss
I0120 10:33:54.308866 140191300101952 af.py:224] donated_invars=donated_invars, inline=inline)
I0120 10:33:54.308907 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/jax/core.py", line 1661, in bind
I0120 10:33:54.308949 140191300101952 af.py:224] return call_bind(self, fun, *args, **params)
I0120 10:33:54.308990 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/jax/core.py", line 1652, in call_bind
I0120 10:33:54.309031 140191300101952 af.py:224] outs = primitive.process(top_trace, fun, tracers, params)
I0120 10:33:54.309072 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/jax/core.py", line 1664, in process
I0120 10:33:54.309114 140191300101952 af.py:224] return trace.process_call(self, fun, tracers, params)
I0120 10:33:54.309155 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/jax/core.py", line 633, in process_call
I0120 10:33:54.309196 140191300101952 af.py:224] return primitive.impl(f, *tracers, **params)
I0120 10:33:54.309239 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/jax/_src/dispatch.py", line 131, in _xla_call_impl
I0120 10:33:54.309290 140191300101952 af.py:224] out = compiled_fun(*args)
I0120 10:33:54.309333 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/jax/_src/dispatch.py", line 436, in _execute_compiled
I0120 10:33:54.309392 140191300101952 af.py:224] out_bufs = compiled.execute(input_bufs)
I0120 10:33:54.309434 140191300101952 af.py:224] jax._src.traceback_util.UnfilteredStackTrace: RuntimeError: INTERNAL: Failed to launch CUDA kernel: fusion_189 with block dimensions: 256x1x1 and grid dimensions: 625x1x1: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:54.309491 140191300101952 af.py:224] 
I0120 10:33:54.309533 140191300101952 af.py:224] The stack trace below excludes JAX-internal frames.
I0120 10:33:54.309575 140191300101952 af.py:224] The preceding is the original exception that occurred, unmodified.
I0120 10:33:54.309616 140191300101952 af.py:224] 
I0120 10:33:54.309658 140191300101952 af.py:224] --------------------
I0120 10:33:54.309699 140191300101952 af.py:224] 
I0120 10:33:54.309740 140191300101952 af.py:224] The above exception was the direct cause of the following exception:
I0120 10:33:54.309782 140191300101952 af.py:224] 
I0120 10:33:54.309823 140191300101952 af.py:224] Traceback (most recent call last):
I0120 10:33:54.309864 140191300101952 af.py:224] File "/app/alphafold/run_alphafold.py", line 427, in <module>
I0120 10:33:54.309906 140191300101952 af.py:224] app.run(main)
I0120 10:33:54.309947 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 312, in run
I0120 10:33:54.309988 140191300101952 af.py:224] _run_main(main, args)
I0120 10:33:54.310029 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 258, in _run_main
I0120 10:33:54.310071 140191300101952 af.py:224] sys.exit(main(argv))
I0120 10:33:54.310112 140191300101952 af.py:224] File "/app/alphafold/run_alphafold.py", line 412, in main
I0120 10:33:54.310153 140191300101952 af.py:224] is_prokaryote=is_prokaryote)
I0120 10:33:54.310195 140191300101952 af.py:224] File "/app/alphafold/run_alphafold.py", line 194, in predict_structure
I0120 10:33:54.310236 140191300101952 af.py:224] random_seed=model_random_seed)
I0120 10:33:54.310278 140191300101952 af.py:224] File "/app/alphafold/alphafold/model/model.py", line 167, in predict
I0120 10:33:54.310319 140191300101952 af.py:224] result = self.apply(self.params, jax.random.PRNGKey(random_seed), feat)
I0120 10:33:54.310359 140191300101952 af.py:224] File "/opt/conda/lib/python3.7/site-packages/jax/_src/dispatch.py", line 436, in _execute_compiled
I0120 10:33:54.310400 140191300101952 af.py:224] out_bufs = compiled.execute(input_bufs)
I0120 10:33:54.310441 140191300101952 af.py:224] RuntimeError: INTERNAL: Failed to launch CUDA kernel: fusion_189 with block dimensions: 256x1x1 and grid dimensions: 625x1x1: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.086571 140191300101952 af.py:224] 2022-01-20 18:33:55.082800: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.086693 140191300101952 af.py:224] 2022-01-20 18:33:55.086161: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.086747 140191300101952 af.py:224] 2022-01-20 18:33:55.086169: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.086811 140191300101952 af.py:224] 2022-01-20 18:33:55.086178: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.086856 140191300101952 af.py:224] 2022-01-20 18:33:55.086182: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.086962 140191300101952 af.py:224] 2022-01-20 18:33:55.086195: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.087007 140191300101952 af.py:224] 2022-01-20 18:33:55.086205: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087051 140191300101952 af.py:224] 2022-01-20 18:33:55.086210: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087094 140191300101952 af.py:224] 2022-01-20 18:33:55.086214: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.087137 140191300101952 af.py:224] 2022-01-20 18:33:55.086219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087180 140191300101952 af.py:224] 2022-01-20 18:33:55.086223: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.087221 140191300101952 af.py:224] 2022-01-20 18:33:55.086232: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.087264 140191300101952 af.py:224] 2022-01-20 18:33:55.086239: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087306 140191300101952 af.py:224] 2022-01-20 18:33:55.086245: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087349 140191300101952 af.py:224] 2022-01-20 18:33:55.086250: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.087391 140191300101952 af.py:224] 2022-01-20 18:33:55.086256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087433 140191300101952 af.py:224] 2022-01-20 18:33:55.086260: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.087475 140191300101952 af.py:224] 2022-01-20 18:33:55.086270: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.087517 140191300101952 af.py:224] 2022-01-20 18:33:55.086276: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087558 140191300101952 af.py:224] 2022-01-20 18:33:55.086281: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087600 140191300101952 af.py:224] 2022-01-20 18:33:55.086285: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.087647 140191300101952 af.py:224] 2022-01-20 18:33:55.086291: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087690 140191300101952 af.py:224] 2022-01-20 18:33:55.086295: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.087732 140191300101952 af.py:224] 2022-01-20 18:33:55.086303: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.087774 140191300101952 af.py:224] 2022-01-20 18:33:55.086309: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087816 140191300101952 af.py:224] 2022-01-20 18:33:55.086313: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087859 140191300101952 af.py:224] 2022-01-20 18:33:55.086317: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.087901 140191300101952 af.py:224] 2022-01-20 18:33:55.086322: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.087943 140191300101952 af.py:224] 2022-01-20 18:33:55.086351: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.087985 140191300101952 af.py:224] 2022-01-20 18:33:55.086359: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.088026 140191300101952 af.py:224] 2022-01-20 18:33:55.086365: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088068 140191300101952 af.py:224] 2022-01-20 18:33:55.086370: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088109 140191300101952 af.py:224] 2022-01-20 18:33:55.086374: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.088166 140191300101952 af.py:224] 2022-01-20 18:33:55.086396: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088208 140191300101952 af.py:224] 2022-01-20 18:33:55.086400: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.088251 140191300101952 af.py:224] 2022-01-20 18:33:55.086426: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.088292 140191300101952 af.py:224] 2022-01-20 18:33:55.086432: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088334 140191300101952 af.py:224] 2022-01-20 18:33:55.086438: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088380 140191300101952 af.py:224] 2022-01-20 18:33:55.086443: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.088423 140191300101952 af.py:224] 2022-01-20 18:33:55.086453: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088481 140191300101952 af.py:224] 2022-01-20 18:33:55.086460: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.088524 140191300101952 af.py:224] 2022-01-20 18:33:55.086473: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.088566 140191300101952 af.py:224] 2022-01-20 18:33:55.086481: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088609 140191300101952 af.py:224] 2022-01-20 18:33:55.086487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088651 140191300101952 af.py:224] 2022-01-20 18:33:55.086492: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.088693 140191300101952 af.py:224] 2022-01-20 18:33:55.086499: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088735 140191300101952 af.py:224] 2022-01-20 18:33:55.086522: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.088777 140191300101952 af.py:224] 2022-01-20 18:33:55.086552: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.088818 140191300101952 af.py:224] 2022-01-20 18:33:55.086560: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088860 140191300101952 af.py:224] 2022-01-20 18:33:55.086566: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088902 140191300101952 af.py:224] 2022-01-20 18:33:55.086572: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.088944 140191300101952 af.py:224] 2022-01-20 18:33:55.086858: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.088986 140191300101952 af.py:224] 2022-01-20 18:33:55.086869: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.089028 140191300101952 af.py:224] 2022-01-20 18:33:55.086897: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.089071 140191300101952 af.py:224] 2022-01-20 18:33:55.086925: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089119 140191300101952 af.py:224] 2022-01-20 18:33:55.086931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089162 140191300101952 af.py:224] 2022-01-20 18:33:55.086937: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.089204 140191300101952 af.py:224] 2022-01-20 18:33:55.086943: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089246 140191300101952 af.py:224] 2022-01-20 18:33:55.086949: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.089288 140191300101952 af.py:224] 2022-01-20 18:33:55.086957: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.089330 140191300101952 af.py:224] 2022-01-20 18:33:55.086964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089372 140191300101952 af.py:224] 2022-01-20 18:33:55.086971: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089414 140191300101952 af.py:224] 2022-01-20 18:33:55.086976: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.089457 140191300101952 af.py:224] 2022-01-20 18:33:55.086983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089499 140191300101952 af.py:224] 2022-01-20 18:33:55.086988: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.089542 140191300101952 af.py:224] 2022-01-20 18:33:55.086997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.089584 140191300101952 af.py:224] 2022-01-20 18:33:55.087004: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089625 140191300101952 af.py:224] 2022-01-20 18:33:55.087026: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089667 140191300101952 af.py:224] 2022-01-20 18:33:55.087032: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.089709 140191300101952 af.py:224] 2022-01-20 18:33:55.087038: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089751 140191300101952 af.py:224] 2022-01-20 18:33:55.087043: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.089793 140191300101952 af.py:224] 2022-01-20 18:33:55.087052: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.089838 140191300101952 af.py:224] 2022-01-20 18:33:55.087059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089881 140191300101952 af.py:224] 2022-01-20 18:33:55.087065: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.089923 140191300101952 af.py:224] 2022-01-20 18:33:55.087070: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.089965 140191300101952 af.py:224] 2022-01-20 18:33:55.087077: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090007 140191300101952 af.py:224] 2022-01-20 18:33:55.087082: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.090050 140191300101952 af.py:224] 2022-01-20 18:33:55.087092: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.090093 140191300101952 af.py:224] 2022-01-20 18:33:55.087098: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090153 140191300101952 af.py:224] 2022-01-20 18:33:55.087104: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090197 140191300101952 af.py:224] 2022-01-20 18:33:55.087109: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.090240 140191300101952 af.py:224] 2022-01-20 18:33:55.087116: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090282 140191300101952 af.py:224] 2022-01-20 18:33:55.087121: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.090324 140191300101952 af.py:224] 2022-01-20 18:33:55.087129: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.090366 140191300101952 af.py:224] 2022-01-20 18:33:55.087136: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090409 140191300101952 af.py:224] 2022-01-20 18:33:55.087142: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090451 140191300101952 af.py:224] 2022-01-20 18:33:55.087147: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.090492 140191300101952 af.py:224] 2022-01-20 18:33:55.087154: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090539 140191300101952 af.py:224] 2022-01-20 18:33:55.087159: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.090581 140191300101952 af.py:224] 2022-01-20 18:33:55.087169: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.090623 140191300101952 af.py:224] 2022-01-20 18:33:55.087176: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090665 140191300101952 af.py:224] 2022-01-20 18:33:55.087182: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090708 140191300101952 af.py:224] 2022-01-20 18:33:55.087187: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.090750 140191300101952 af.py:224] 2022-01-20 18:33:55.087193: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090793 140191300101952 af.py:224] 2022-01-20 18:33:55.087199: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.090834 140191300101952 af.py:224] 2022-01-20 18:33:55.087207: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.090876 140191300101952 af.py:224] 2022-01-20 18:33:55.087214: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090919 140191300101952 af.py:224] 2022-01-20 18:33:55.087220: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.090960 140191300101952 af.py:224] 2022-01-20 18:33:55.087225: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.091002 140191300101952 af.py:224] 2022-01-20 18:33:55.087231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091043 140191300101952 af.py:224] 2022-01-20 18:33:55.087236: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.091086 140191300101952 af.py:224] 2022-01-20 18:33:55.087245: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.091128 140191300101952 af.py:224] 2022-01-20 18:33:55.087251: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091170 140191300101952 af.py:224] 2022-01-20 18:33:55.087257: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091212 140191300101952 af.py:224] 2022-01-20 18:33:55.087262: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.091257 140191300101952 af.py:224] 2022-01-20 18:33:55.087269: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091299 140191300101952 af.py:224] 2022-01-20 18:33:55.087274: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.091341 140191300101952 af.py:224] 2022-01-20 18:33:55.087282: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.091384 140191300101952 af.py:224] 2022-01-20 18:33:55.087303: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091426 140191300101952 af.py:224] 2022-01-20 18:33:55.087310: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091467 140191300101952 af.py:224] 2022-01-20 18:33:55.087316: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.091509 140191300101952 af.py:224] 2022-01-20 18:33:55.087322: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091551 140191300101952 af.py:224] 2022-01-20 18:33:55.087328: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.091592 140191300101952 af.py:224] 2022-01-20 18:33:55.087337: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.091634 140191300101952 af.py:224] 2022-01-20 18:33:55.087345: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091676 140191300101952 af.py:224] 2022-01-20 18:33:55.087352: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091718 140191300101952 af.py:224] 2022-01-20 18:33:55.087357: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.091759 140191300101952 af.py:224] 2022-01-20 18:33:55.087363: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091818 140191300101952 af.py:224] 2022-01-20 18:33:55.087368: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.091861 140191300101952 af.py:224] 2022-01-20 18:33:55.087377: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.091903 140191300101952 af.py:224] 2022-01-20 18:33:55.087385: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091946 140191300101952 af.py:224] 2022-01-20 18:33:55.087391: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.091993 140191300101952 af.py:224] 2022-01-20 18:33:55.087398: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.092035 140191300101952 af.py:224] 2022-01-20 18:33:55.087408: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092076 140191300101952 af.py:224] 2022-01-20 18:33:55.087418: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.092121 140191300101952 af.py:224] 2022-01-20 18:33:55.087432: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.092165 140191300101952 af.py:224] 2022-01-20 18:33:55.087446: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092207 140191300101952 af.py:224] 2022-01-20 18:33:55.087457: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092249 140191300101952 af.py:224] 2022-01-20 18:33:55.087467: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.092292 140191300101952 af.py:224] 2022-01-20 18:33:55.087479: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092334 140191300101952 af.py:224] 2022-01-20 18:33:55.087488: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.092377 140191300101952 af.py:224] 2022-01-20 18:33:55.087504: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.092419 140191300101952 af.py:224] 2022-01-20 18:33:55.087517: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092461 140191300101952 af.py:224] 2022-01-20 18:33:55.087528: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092503 140191300101952 af.py:224] 2022-01-20 18:33:55.087536: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.092545 140191300101952 af.py:224] 2022-01-20 18:33:55.087545: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092586 140191300101952 af.py:224] 2022-01-20 18:33:55.087550: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.092628 140191300101952 af.py:224] 2022-01-20 18:33:55.087560: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.092669 140191300101952 af.py:224] 2022-01-20 18:33:55.087568: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092715 140191300101952 af.py:224] 2022-01-20 18:33:55.087575: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092757 140191300101952 af.py:224] 2022-01-20 18:33:55.087580: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.092799 140191300101952 af.py:224] 2022-01-20 18:33:55.087588: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092842 140191300101952 af.py:224] 2022-01-20 18:33:55.087597: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.092884 140191300101952 af.py:224] 2022-01-20 18:33:55.087612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.092926 140191300101952 af.py:224] 2022-01-20 18:33:55.087623: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.092967 140191300101952 af.py:224] 2022-01-20 18:33:55.087633: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093009 140191300101952 af.py:224] 2022-01-20 18:33:55.087642: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.093051 140191300101952 af.py:224] 2022-01-20 18:33:55.087654: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093094 140191300101952 af.py:224] 2022-01-20 18:33:55.087663: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.093136 140191300101952 af.py:224] 2022-01-20 18:33:55.087678: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.093178 140191300101952 af.py:224] 2022-01-20 18:33:55.087690: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093220 140191300101952 af.py:224] 2022-01-20 18:33:55.087718: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093262 140191300101952 af.py:224] 2022-01-20 18:33:55.087728: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.093303 140191300101952 af.py:224] 2022-01-20 18:33:55.087736: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093345 140191300101952 af.py:224] 2022-01-20 18:33:55.087741: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.093386 140191300101952 af.py:224] 2022-01-20 18:33:55.087751: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.093431 140191300101952 af.py:224] 2022-01-20 18:33:55.087758: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093491 140191300101952 af.py:224] 2022-01-20 18:33:55.087764: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093534 140191300101952 af.py:224] 2022-01-20 18:33:55.087769: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.093576 140191300101952 af.py:224] 2022-01-20 18:33:55.087777: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093618 140191300101952 af.py:224] 2022-01-20 18:33:55.087784: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.093660 140191300101952 af.py:224] 2022-01-20 18:33:55.087797: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.093702 140191300101952 af.py:224] 2022-01-20 18:33:55.087805: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093745 140191300101952 af.py:224] 2022-01-20 18:33:55.087811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093787 140191300101952 af.py:224] 2022-01-20 18:33:55.087816: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.093829 140191300101952 af.py:224] 2022-01-20 18:33:55.087823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093871 140191300101952 af.py:224] 2022-01-20 18:33:55.087828: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.093913 140191300101952 af.py:224] 2022-01-20 18:33:55.087840: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.093955 140191300101952 af.py:224] 2022-01-20 18:33:55.087850: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.093998 140191300101952 af.py:224] 2022-01-20 18:33:55.087857: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094040 140191300101952 af.py:224] 2022-01-20 18:33:55.087862: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.094082 140191300101952 af.py:224] 2022-01-20 18:33:55.087869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094128 140191300101952 af.py:224] 2022-01-20 18:33:55.087874: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.094170 140191300101952 af.py:224] 2022-01-20 18:33:55.087883: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.094212 140191300101952 af.py:224] 2022-01-20 18:33:55.087890: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094255 140191300101952 af.py:224] 2022-01-20 18:33:55.087898: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094297 140191300101952 af.py:224] 2022-01-20 18:33:55.087907: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.094339 140191300101952 af.py:224] 2022-01-20 18:33:55.087919: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094382 140191300101952 af.py:224] 2022-01-20 18:33:55.087928: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.094424 140191300101952 af.py:224] 2022-01-20 18:33:55.087943: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.094466 140191300101952 af.py:224] 2022-01-20 18:33:55.087956: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094509 140191300101952 af.py:224] 2022-01-20 18:33:55.087964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094551 140191300101952 af.py:224] 2022-01-20 18:33:55.087970: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.094592 140191300101952 af.py:224] 2022-01-20 18:33:55.087977: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094634 140191300101952 af.py:224] 2022-01-20 18:33:55.087982: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.094676 140191300101952 af.py:224] 2022-01-20 18:33:55.087992: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.094717 140191300101952 af.py:224] 2022-01-20 18:33:55.088000: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094760 140191300101952 af.py:224] 2022-01-20 18:33:55.088008: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094802 140191300101952 af.py:224] 2022-01-20 18:33:55.088015: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.094848 140191300101952 af.py:224] 2022-01-20 18:33:55.088023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.094908 140191300101952 af.py:224] 2022-01-20 18:33:55.088028: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.094975 140191300101952 af.py:224] 2022-01-20 18:33:55.088037: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.095043 140191300101952 af.py:224] 2022-01-20 18:33:55.088044: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095111 140191300101952 af.py:224] 2022-01-20 18:33:55.088050: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095180 140191300101952 af.py:224] 2022-01-20 18:33:55.088055: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.095229 140191300101952 af.py:224] 2022-01-20 18:33:55.088062: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095273 140191300101952 af.py:224] 2022-01-20 18:33:55.088069: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.095334 140191300101952 af.py:224] 2022-01-20 18:33:55.088085: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.095378 140191300101952 af.py:224] 2022-01-20 18:33:55.088097: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095421 140191300101952 af.py:224] 2022-01-20 18:33:55.088108: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095482 140191300101952 af.py:224] 2022-01-20 18:33:55.088122: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.095533 140191300101952 af.py:224] 2022-01-20 18:33:55.088135: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095577 140191300101952 af.py:224] 2022-01-20 18:33:55.088144: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.095619 140191300101952 af.py:224] 2022-01-20 18:33:55.088158: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.095663 140191300101952 af.py:224] 2022-01-20 18:33:55.088171: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095706 140191300101952 af.py:224] 2022-01-20 18:33:55.088180: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095754 140191300101952 af.py:224] 2022-01-20 18:33:55.088189: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.095797 140191300101952 af.py:224] 2022-01-20 18:33:55.088201: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095840 140191300101952 af.py:224] 2022-01-20 18:33:55.088210: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.095884 140191300101952 af.py:224] 2022-01-20 18:33:55.088226: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.095927 140191300101952 af.py:224] 2022-01-20 18:33:55.088239: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.095970 140191300101952 af.py:224] 2022-01-20 18:33:55.088249: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096013 140191300101952 af.py:224] 2022-01-20 18:33:55.088258: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.096055 140191300101952 af.py:224] 2022-01-20 18:33:55.088270: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096097 140191300101952 af.py:224] 2022-01-20 18:33:55.088279: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.096145 140191300101952 af.py:224] 2022-01-20 18:33:55.088312: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.096189 140191300101952 af.py:224] 2022-01-20 18:33:55.088323: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096231 140191300101952 af.py:224] 2022-01-20 18:33:55.088333: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096273 140191300101952 af.py:224] 2022-01-20 18:33:55.088339: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.096315 140191300101952 af.py:224] 2022-01-20 18:33:55.088345: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096357 140191300101952 af.py:224] 2022-01-20 18:33:55.088350: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.096398 140191300101952 af.py:224] 2022-01-20 18:33:55.088360: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.096440 140191300101952 af.py:224] 2022-01-20 18:33:55.088367: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096486 140191300101952 af.py:224] 2022-01-20 18:33:55.088373: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096528 140191300101952 af.py:224] 2022-01-20 18:33:55.088378: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.096571 140191300101952 af.py:224] 2022-01-20 18:33:55.088387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096613 140191300101952 af.py:224] 2022-01-20 18:33:55.088397: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.096655 140191300101952 af.py:224] 2022-01-20 18:33:55.088410: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.096698 140191300101952 af.py:224] 2022-01-20 18:33:55.088418: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096739 140191300101952 af.py:224] 2022-01-20 18:33:55.088424: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096781 140191300101952 af.py:224] 2022-01-20 18:33:55.088430: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.096824 140191300101952 af.py:224] 2022-01-20 18:33:55.088436: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.096866 140191300101952 af.py:224] 2022-01-20 18:33:55.088441: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.096908 140191300101952 af.py:224] 2022-01-20 18:33:55.088451: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.096950 140191300101952 af.py:224] 2022-01-20 18:33:55.088460: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097010 140191300101952 af.py:224] 2022-01-20 18:33:55.088469: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097054 140191300101952 af.py:224] 2022-01-20 18:33:55.088474: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.097097 140191300101952 af.py:224] 2022-01-20 18:33:55.088481: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097139 140191300101952 af.py:224] 2022-01-20 18:33:55.088486: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.097181 140191300101952 af.py:224] 2022-01-20 18:33:55.088495: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.097227 140191300101952 af.py:224] 2022-01-20 18:33:55.088502: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097270 140191300101952 af.py:224] 2022-01-20 18:33:55.088509: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097312 140191300101952 af.py:224] 2022-01-20 18:33:55.088514: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.097354 140191300101952 af.py:224] 2022-01-20 18:33:55.088523: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097396 140191300101952 af.py:224] 2022-01-20 18:33:55.088532: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.097439 140191300101952 af.py:224] 2022-01-20 18:33:55.088546: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.097481 140191300101952 af.py:224] 2022-01-20 18:33:55.088555: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097523 140191300101952 af.py:224] 2022-01-20 18:33:55.088561: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097565 140191300101952 af.py:224] 2022-01-20 18:33:55.088567: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.097607 140191300101952 af.py:224] 2022-01-20 18:33:55.088573: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097649 140191300101952 af.py:224] 2022-01-20 18:33:55.088578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.097692 140191300101952 af.py:224] 2022-01-20 18:33:55.088587: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.097733 140191300101952 af.py:224] 2022-01-20 18:33:55.088595: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097776 140191300101952 af.py:224] 2022-01-20 18:33:55.088605: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097818 140191300101952 af.py:224] 2022-01-20 18:33:55.088615: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.097862 140191300101952 af.py:224] 2022-01-20 18:33:55.088626: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.097907 140191300101952 af.py:224] 2022-01-20 18:33:55.088634: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.097950 140191300101952 af.py:224] 2022-01-20 18:33:55.088647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.097992 140191300101952 af.py:224] 2022-01-20 18:33:55.088656: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098034 140191300101952 af.py:224] 2022-01-20 18:33:55.088663: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098077 140191300101952 af.py:224] 2022-01-20 18:33:55.088669: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.098119 140191300101952 af.py:224] 2022-01-20 18:33:55.088677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098162 140191300101952 af.py:224] 2022-01-20 18:33:55.088683: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.098204 140191300101952 af.py:224] 2022-01-20 18:33:55.088694: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.098246 140191300101952 af.py:224] 2022-01-20 18:33:55.088702: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098289 140191300101952 af.py:224] 2022-01-20 18:33:55.088709: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098331 140191300101952 af.py:224] 2022-01-20 18:33:55.088715: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.098373 140191300101952 af.py:224] 2022-01-20 18:33:55.088723: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098415 140191300101952 af.py:224] 2022-01-20 18:33:55.088729: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.098456 140191300101952 af.py:224] 2022-01-20 18:33:55.088739: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.098498 140191300101952 af.py:224] 2022-01-20 18:33:55.088747: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098540 140191300101952 af.py:224] 2022-01-20 18:33:55.088755: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098582 140191300101952 af.py:224] 2022-01-20 18:33:55.088760: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.098627 140191300101952 af.py:224] 2022-01-20 18:33:55.088768: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098670 140191300101952 af.py:224] 2022-01-20 18:33:55.088774: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.098728 140191300101952 af.py:224] 2022-01-20 18:33:55.088785: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.098772 140191300101952 af.py:224] 2022-01-20 18:33:55.088792: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098814 140191300101952 af.py:224] 2022-01-20 18:33:55.088799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098856 140191300101952 af.py:224] 2022-01-20 18:33:55.088805: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.098898 140191300101952 af.py:224] 2022-01-20 18:33:55.088813: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.098940 140191300101952 af.py:224] 2022-01-20 18:33:55.088819: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.098983 140191300101952 af.py:224] 2022-01-20 18:33:55.088829: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.099025 140191300101952 af.py:224] 2022-01-20 18:33:55.088856: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099067 140191300101952 af.py:224] 2022-01-20 18:33:55.088867: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099110 140191300101952 af.py:224] 2022-01-20 18:33:55.088876: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.099151 140191300101952 af.py:224] 2022-01-20 18:33:55.088888: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099194 140191300101952 af.py:224] 2022-01-20 18:33:55.088897: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.099236 140191300101952 af.py:224] 2022-01-20 18:33:55.088913: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.099277 140191300101952 af.py:224] 2022-01-20 18:33:55.088925: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099320 140191300101952 af.py:224] 2022-01-20 18:33:55.088935: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099366 140191300101952 af.py:224] 2022-01-20 18:33:55.088944: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.099408 140191300101952 af.py:224] 2022-01-20 18:33:55.088956: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099450 140191300101952 af.py:224] 2022-01-20 18:33:55.088965: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.099493 140191300101952 af.py:224] 2022-01-20 18:33:55.088981: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.099535 140191300101952 af.py:224] 2022-01-20 18:33:55.088995: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099577 140191300101952 af.py:224] 2022-01-20 18:33:55.089007: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099619 140191300101952 af.py:224] 2022-01-20 18:33:55.089015: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.099661 140191300101952 af.py:224] 2022-01-20 18:33:55.089022: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099703 140191300101952 af.py:224] 2022-01-20 18:33:55.089028: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.099745 140191300101952 af.py:224] 2022-01-20 18:33:55.089038: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.099787 140191300101952 af.py:224] 2022-01-20 18:33:55.089046: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099829 140191300101952 af.py:224] 2022-01-20 18:33:55.089052: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099872 140191300101952 af.py:224] 2022-01-20 18:33:55.089057: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.099913 140191300101952 af.py:224] 2022-01-20 18:33:55.089064: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.099955 140191300101952 af.py:224] 2022-01-20 18:33:55.089069: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.099997 140191300101952 af.py:224] 2022-01-20 18:33:55.089079: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.100038 140191300101952 af.py:224] 2022-01-20 18:33:55.089086: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100085 140191300101952 af.py:224] 2022-01-20 18:33:55.089093: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100132 140191300101952 af.py:224] 2022-01-20 18:33:55.089098: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.100176 140191300101952 af.py:224] 2022-01-20 18:33:55.089104: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100219 140191300101952 af.py:224] 2022-01-20 18:33:55.089109: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.100261 140191300101952 af.py:224] 2022-01-20 18:33:55.089120: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.100303 140191300101952 af.py:224] 2022-01-20 18:33:55.089130: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100346 140191300101952 af.py:224] 2022-01-20 18:33:55.089137: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100405 140191300101952 af.py:224] 2022-01-20 18:33:55.089142: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.100448 140191300101952 af.py:224] 2022-01-20 18:33:55.089149: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100491 140191300101952 af.py:224] 2022-01-20 18:33:55.089154: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.100533 140191300101952 af.py:224] 2022-01-20 18:33:55.089163: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.100574 140191300101952 af.py:224] 2022-01-20 18:33:55.089172: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100617 140191300101952 af.py:224] 2022-01-20 18:33:55.089179: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100660 140191300101952 af.py:224] 2022-01-20 18:33:55.089187: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.100701 140191300101952 af.py:224] 2022-01-20 18:33:55.089199: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100744 140191300101952 af.py:224] 2022-01-20 18:33:55.089208: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.100785 140191300101952 af.py:224] 2022-01-20 18:33:55.089223: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.100830 140191300101952 af.py:224] 2022-01-20 18:33:55.089236: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100872 140191300101952 af.py:224] 2022-01-20 18:33:55.089246: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100914 140191300101952 af.py:224] 2022-01-20 18:33:55.089256: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.100956 140191300101952 af.py:224] 2022-01-20 18:33:55.089269: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.100998 140191300101952 af.py:224] 2022-01-20 18:33:55.089279: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.101041 140191300101952 af.py:224] 2022-01-20 18:33:55.089300: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.101083 140191300101952 af.py:224] 2022-01-20 18:33:55.089313: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101125 140191300101952 af.py:224] 2022-01-20 18:33:55.089323: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101167 140191300101952 af.py:224] 2022-01-20 18:33:55.089333: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.101209 140191300101952 af.py:224] 2022-01-20 18:33:55.089346: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101251 140191300101952 af.py:224] 2022-01-20 18:33:55.089356: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.101293 140191300101952 af.py:224] 2022-01-20 18:33:55.089371: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.101335 140191300101952 af.py:224] 2022-01-20 18:33:55.089382: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101377 140191300101952 af.py:224] 2022-01-20 18:33:55.089393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101420 140191300101952 af.py:224] 2022-01-20 18:33:55.089401: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.101461 140191300101952 af.py:224] 2022-01-20 18:33:55.089409: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101507 140191300101952 af.py:224] 2022-01-20 18:33:55.089414: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.101549 140191300101952 af.py:224] 2022-01-20 18:33:55.089423: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.101591 140191300101952 af.py:224] 2022-01-20 18:33:55.089431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101633 140191300101952 af.py:224] 2022-01-20 18:33:55.089438: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101675 140191300101952 af.py:224] 2022-01-20 18:33:55.089443: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.101717 140191300101952 af.py:224] 2022-01-20 18:33:55.089451: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101760 140191300101952 af.py:224] 2022-01-20 18:33:55.089460: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.101802 140191300101952 af.py:224] 2022-01-20 18:33:55.089475: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.101844 140191300101952 af.py:224] 2022-01-20 18:33:55.089488: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101886 140191300101952 af.py:224] 2022-01-20 18:33:55.089499: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.101927 140191300101952 af.py:224] 2022-01-20 18:33:55.089509: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.101970 140191300101952 af.py:224] 2022-01-20 18:33:55.089522: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102012 140191300101952 af.py:224] 2022-01-20 18:33:55.089532: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.102071 140191300101952 af.py:224] 2022-01-20 18:33:55.089548: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.102114 140191300101952 af.py:224] 2022-01-20 18:33:55.089563: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102157 140191300101952 af.py:224] 2022-01-20 18:33:55.089573: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102200 140191300101952 af.py:224] 2022-01-20 18:33:55.089582: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.102246 140191300101952 af.py:224] 2022-01-20 18:33:55.089592: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102289 140191300101952 af.py:224] 2022-01-20 18:33:55.089599: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.102331 140191300101952 af.py:224] 2022-01-20 18:33:55.089613: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.102374 140191300101952 af.py:224] 2022-01-20 18:33:55.089629: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102416 140191300101952 af.py:224] 2022-01-20 18:33:55.089639: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102458 140191300101952 af.py:224] 2022-01-20 18:33:55.089648: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.102500 140191300101952 af.py:224] 2022-01-20 18:33:55.089660: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102543 140191300101952 af.py:224] 2022-01-20 18:33:55.089670: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.102586 140191300101952 af.py:224] 2022-01-20 18:33:55.089686: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.102628 140191300101952 af.py:224] 2022-01-20 18:33:55.089699: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102670 140191300101952 af.py:224] 2022-01-20 18:33:55.089710: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102712 140191300101952 af.py:224] 2022-01-20 18:33:55.089718: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.102754 140191300101952 af.py:224] 2022-01-20 18:33:55.089726: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102796 140191300101952 af.py:224] 2022-01-20 18:33:55.089733: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.102838 140191300101952 af.py:224] 2022-01-20 18:33:55.089744: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.102880 140191300101952 af.py:224] 2022-01-20 18:33:55.089752: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102923 140191300101952 af.py:224] 2022-01-20 18:33:55.089759: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.102968 140191300101952 af.py:224] 2022-01-20 18:33:55.089764: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.103011 140191300101952 af.py:224] 2022-01-20 18:33:55.089771: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103053 140191300101952 af.py:224] 2022-01-20 18:33:55.089777: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.103095 140191300101952 af.py:224] 2022-01-20 18:33:55.089788: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.103136 140191300101952 af.py:224] 2022-01-20 18:33:55.089797: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103178 140191300101952 af.py:224] 2022-01-20 18:33:55.089803: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103220 140191300101952 af.py:224] 2022-01-20 18:33:55.089808: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.103262 140191300101952 af.py:224] 2022-01-20 18:33:55.089815: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103304 140191300101952 af.py:224] 2022-01-20 18:33:55.089821: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.103347 140191300101952 af.py:224] 2022-01-20 18:33:55.089829: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.103389 140191300101952 af.py:224] 2022-01-20 18:33:55.089836: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103431 140191300101952 af.py:224] 2022-01-20 18:33:55.089842: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103473 140191300101952 af.py:224] 2022-01-20 18:33:55.089848: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.103515 140191300101952 af.py:224] 2022-01-20 18:33:55.089855: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103557 140191300101952 af.py:224] 2022-01-20 18:33:55.089860: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.103600 140191300101952 af.py:224] 2022-01-20 18:33:55.089868: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.103641 140191300101952 af.py:224] 2022-01-20 18:33:55.089875: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103686 140191300101952 af.py:224] 2022-01-20 18:33:55.089882: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103745 140191300101952 af.py:224] 2022-01-20 18:33:55.089887: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.103788 140191300101952 af.py:224] 2022-01-20 18:33:55.089896: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103831 140191300101952 af.py:224] 2022-01-20 18:33:55.089902: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.103873 140191300101952 af.py:224] 2022-01-20 18:33:55.089930: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.103915 140191300101952 af.py:224] 2022-01-20 18:33:55.089940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103957 140191300101952 af.py:224] 2022-01-20 18:33:55.089949: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.103999 140191300101952 af.py:224] 2022-01-20 18:33:55.089958: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.104041 140191300101952 af.py:224] 2022-01-20 18:33:55.089970: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104083 140191300101952 af.py:224] 2022-01-20 18:33:55.089979: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.104130 140191300101952 af.py:224] 2022-01-20 18:33:55.089993: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.104174 140191300101952 af.py:224] 2022-01-20 18:33:55.090006: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104217 140191300101952 af.py:224] 2022-01-20 18:33:55.090016: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104259 140191300101952 af.py:224] 2022-01-20 18:33:55.090025: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.104301 140191300101952 af.py:224] 2022-01-20 18:33:55.090038: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104343 140191300101952 af.py:224] 2022-01-20 18:33:55.090047: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.104386 140191300101952 af.py:224] 2022-01-20 18:33:55.090061: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.104431 140191300101952 af.py:224] 2022-01-20 18:33:55.090074: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104474 140191300101952 af.py:224] 2022-01-20 18:33:55.090082: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104516 140191300101952 af.py:224] 2022-01-20 18:33:55.090088: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.104557 140191300101952 af.py:224] 2022-01-20 18:33:55.090095: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104600 140191300101952 af.py:224] 2022-01-20 18:33:55.090100: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.104642 140191300101952 af.py:224] 2022-01-20 18:33:55.090109: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.104684 140191300101952 af.py:224] 2022-01-20 18:33:55.090117: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104726 140191300101952 af.py:224] 2022-01-20 18:33:55.090124: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104769 140191300101952 af.py:224] 2022-01-20 18:33:55.090129: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.104811 140191300101952 af.py:224] 2022-01-20 18:33:55.090138: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104853 140191300101952 af.py:224] 2022-01-20 18:33:55.090146: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.104895 140191300101952 af.py:224] 2022-01-20 18:33:55.090156: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.104937 140191300101952 af.py:224] 2022-01-20 18:33:55.090168: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.104979 140191300101952 af.py:224] 2022-01-20 18:33:55.090179: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105021 140191300101952 af.py:224] 2022-01-20 18:33:55.090187: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.105063 140191300101952 af.py:224] 2022-01-20 18:33:55.090196: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105112 140191300101952 af.py:224] 2022-01-20 18:33:55.090206: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.105177 140191300101952 af.py:224] 2022-01-20 18:33:55.090221: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.105223 140191300101952 af.py:224] 2022-01-20 18:33:55.090232: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105266 140191300101952 af.py:224] 2022-01-20 18:33:55.090242: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105309 140191300101952 af.py:224] 2022-01-20 18:33:55.090252: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.105352 140191300101952 af.py:224] 2022-01-20 18:33:55.090264: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105395 140191300101952 af.py:224] 2022-01-20 18:33:55.090273: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.105438 140191300101952 af.py:224] 2022-01-20 18:33:55.090288: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.105496 140191300101952 af.py:224] 2022-01-20 18:33:55.090301: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105540 140191300101952 af.py:224] 2022-01-20 18:33:55.090313: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105581 140191300101952 af.py:224] 2022-01-20 18:33:55.090323: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.105623 140191300101952 af.py:224] 2022-01-20 18:33:55.090336: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105665 140191300101952 af.py:224] 2022-01-20 18:33:55.090346: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.105707 140191300101952 af.py:224] 2022-01-20 18:33:55.090357: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.105749 140191300101952 af.py:224] 2022-01-20 18:33:55.090365: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105791 140191300101952 af.py:224] 2022-01-20 18:33:55.090372: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105833 140191300101952 af.py:224] 2022-01-20 18:33:55.090377: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.105879 140191300101952 af.py:224] 2022-01-20 18:33:55.090384: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.105922 140191300101952 af.py:224] 2022-01-20 18:33:55.090389: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.105965 140191300101952 af.py:224] 2022-01-20 18:33:55.090398: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.106007 140191300101952 af.py:224] 2022-01-20 18:33:55.090408: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106049 140191300101952 af.py:224] 2022-01-20 18:33:55.090415: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106091 140191300101952 af.py:224] 2022-01-20 18:33:55.090420: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.106133 140191300101952 af.py:224] 2022-01-20 18:33:55.090426: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106176 140191300101952 af.py:224] 2022-01-20 18:33:55.090432: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.106218 140191300101952 af.py:224] 2022-01-20 18:33:55.090440: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.106261 140191300101952 af.py:224] 2022-01-20 18:33:55.090447: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106302 140191300101952 af.py:224] 2022-01-20 18:33:55.090453: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106344 140191300101952 af.py:224] 2022-01-20 18:33:55.090459: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.106386 140191300101952 af.py:224] 2022-01-20 18:33:55.090465: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106428 140191300101952 af.py:224] 2022-01-20 18:33:55.090471: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.106470 140191300101952 af.py:224] 2022-01-20 18:33:55.090480: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.106512 140191300101952 af.py:224] 2022-01-20 18:33:55.090487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106554 140191300101952 af.py:224] 2022-01-20 18:33:55.090493: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106600 140191300101952 af.py:224] 2022-01-20 18:33:55.090498: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.106643 140191300101952 af.py:224] 2022-01-20 18:33:55.090505: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106686 140191300101952 af.py:224] 2022-01-20 18:33:55.090511: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.106728 140191300101952 af.py:224] 2022-01-20 18:33:55.090519: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.106770 140191300101952 af.py:224] 2022-01-20 18:33:55.090529: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106812 140191300101952 af.py:224] 2022-01-20 18:33:55.090536: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106854 140191300101952 af.py:224] 2022-01-20 18:33:55.090541: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.106896 140191300101952 af.py:224] 2022-01-20 18:33:55.090548: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.106938 140191300101952 af.py:224] 2022-01-20 18:33:55.090553: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.106980 140191300101952 af.py:224] 2022-01-20 18:33:55.090576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.107021 140191300101952 af.py:224] 2022-01-20 18:33:55.090584: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107063 140191300101952 af.py:224] 2022-01-20 18:33:55.090591: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107105 140191300101952 af.py:224] 2022-01-20 18:33:55.090596: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.107161 140191300101952 af.py:224] 2022-01-20 18:33:55.090602: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107205 140191300101952 af.py:224] 2022-01-20 18:33:55.090609: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.107247 140191300101952 af.py:224] 2022-01-20 18:33:55.090619: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.107289 140191300101952 af.py:224] 2022-01-20 18:33:55.090627: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107336 140191300101952 af.py:224] 2022-01-20 18:33:55.090634: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107378 140191300101952 af.py:224] 2022-01-20 18:33:55.090639: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.107421 140191300101952 af.py:224] 2022-01-20 18:33:55.090648: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107463 140191300101952 af.py:224] 2022-01-20 18:33:55.090654: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.107506 140191300101952 af.py:224] 2022-01-20 18:33:55.090665: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.107547 140191300101952 af.py:224] 2022-01-20 18:33:55.090674: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107589 140191300101952 af.py:224] 2022-01-20 18:33:55.090680: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107631 140191300101952 af.py:224] 2022-01-20 18:33:55.090686: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.107673 140191300101952 af.py:224] 2022-01-20 18:33:55.090695: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107715 140191300101952 af.py:224] 2022-01-20 18:33:55.090704: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.107756 140191300101952 af.py:224] 2022-01-20 18:33:55.090720: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.107798 140191300101952 af.py:224] 2022-01-20 18:33:55.090733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107840 140191300101952 af.py:224] 2022-01-20 18:33:55.090741: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107882 140191300101952 af.py:224] 2022-01-20 18:33:55.090746: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.107924 140191300101952 af.py:224] 2022-01-20 18:33:55.090753: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.107966 140191300101952 af.py:224] 2022-01-20 18:33:55.090758: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.108008 140191300101952 af.py:224] 2022-01-20 18:33:55.090768: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.108053 140191300101952 af.py:224] 2022-01-20 18:33:55.090775: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108095 140191300101952 af.py:224] 2022-01-20 18:33:55.090782: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108143 140191300101952 af.py:224] 2022-01-20 18:33:55.090791: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.108186 140191300101952 af.py:224] 2022-01-20 18:33:55.090802: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108229 140191300101952 af.py:224] 2022-01-20 18:33:55.090811: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.108271 140191300101952 af.py:224] 2022-01-20 18:33:55.090822: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.108313 140191300101952 af.py:224] 2022-01-20 18:33:55.090830: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108355 140191300101952 af.py:224] 2022-01-20 18:33:55.090837: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108397 140191300101952 af.py:224] 2022-01-20 18:33:55.090842: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.108439 140191300101952 af.py:224] 2022-01-20 18:33:55.090849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108482 140191300101952 af.py:224] 2022-01-20 18:33:55.090854: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.108524 140191300101952 af.py:224] 2022-01-20 18:33:55.090862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.108566 140191300101952 af.py:224] 2022-01-20 18:33:55.090869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108608 140191300101952 af.py:224] 2022-01-20 18:33:55.090875: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108651 140191300101952 af.py:224] 2022-01-20 18:33:55.090880: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.108694 140191300101952 af.py:224] 2022-01-20 18:33:55.090887: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108740 140191300101952 af.py:224] 2022-01-20 18:33:55.090892: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.108783 140191300101952 af.py:224] 2022-01-20 18:33:55.090901: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.108845 140191300101952 af.py:224] 2022-01-20 18:33:55.090908: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108889 140191300101952 af.py:224] 2022-01-20 18:33:55.090914: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.108932 140191300101952 af.py:224] 2022-01-20 18:33:55.090919: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.108974 140191300101952 af.py:224] 2022-01-20 18:33:55.090926: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109016 140191300101952 af.py:224] 2022-01-20 18:33:55.090931: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.109059 140191300101952 af.py:224] 2022-01-20 18:33:55.090940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.109101 140191300101952 af.py:224] 2022-01-20 18:33:55.090947: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109144 140191300101952 af.py:224] 2022-01-20 18:33:55.090953: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109187 140191300101952 af.py:224] 2022-01-20 18:33:55.090958: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.109228 140191300101952 af.py:224] 2022-01-20 18:33:55.090964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109271 140191300101952 af.py:224] 2022-01-20 18:33:55.090969: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.109314 140191300101952 af.py:224] 2022-01-20 18:33:55.090977: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.109355 140191300101952 af.py:224] 2022-01-20 18:33:55.090985: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109398 140191300101952 af.py:224] 2022-01-20 18:33:55.090991: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109441 140191300101952 af.py:224] 2022-01-20 18:33:55.090996: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.109487 140191300101952 af.py:224] 2022-01-20 18:33:55.091018: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109529 140191300101952 af.py:224] 2022-01-20 18:33:55.091024: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.109571 140191300101952 af.py:224] 2022-01-20 18:33:55.091036: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.109613 140191300101952 af.py:224] 2022-01-20 18:33:55.091047: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109655 140191300101952 af.py:224] 2022-01-20 18:33:55.091056: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109697 140191300101952 af.py:224] 2022-01-20 18:33:55.091065: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.109739 140191300101952 af.py:224] 2022-01-20 18:33:55.091076: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109781 140191300101952 af.py:224] 2022-01-20 18:33:55.091084: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.109823 140191300101952 af.py:224] 2022-01-20 18:33:55.091098: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.109865 140191300101952 af.py:224] 2022-01-20 18:33:55.091110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109908 140191300101952 af.py:224] 2022-01-20 18:33:55.091120: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.109950 140191300101952 af.py:224] 2022-01-20 18:33:55.091129: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.109992 140191300101952 af.py:224] 2022-01-20 18:33:55.091141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110034 140191300101952 af.py:224] 2022-01-20 18:33:55.091151: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.110075 140191300101952 af.py:224] 2022-01-20 18:33:55.091164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.110117 140191300101952 af.py:224] 2022-01-20 18:33:55.091172: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110159 140191300101952 af.py:224] 2022-01-20 18:33:55.091179: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110204 140191300101952 af.py:224] 2022-01-20 18:33:55.091184: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.110246 140191300101952 af.py:224] 2022-01-20 18:33:55.091191: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110289 140191300101952 af.py:224] 2022-01-20 18:33:55.091196: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.110331 140191300101952 af.py:224] 2022-01-20 18:33:55.091206: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.110373 140191300101952 af.py:224] 2022-01-20 18:33:55.091213: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110415 140191300101952 af.py:224] 2022-01-20 18:33:55.091221: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110457 140191300101952 af.py:224] 2022-01-20 18:33:55.091229: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.110514 140191300101952 af.py:224] 2022-01-20 18:33:55.091238: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110557 140191300101952 af.py:224] 2022-01-20 18:33:55.091244: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.110599 140191300101952 af.py:224] 2022-01-20 18:33:55.091253: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.110641 140191300101952 af.py:224] 2022-01-20 18:33:55.091260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110684 140191300101952 af.py:224] 2022-01-20 18:33:55.091266: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110726 140191300101952 af.py:224] 2022-01-20 18:33:55.091272: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.110769 140191300101952 af.py:224] 2022-01-20 18:33:55.091281: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110811 140191300101952 af.py:224] 2022-01-20 18:33:55.091286: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.110853 140191300101952 af.py:224] 2022-01-20 18:33:55.091295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.110896 140191300101952 af.py:224] 2022-01-20 18:33:55.091303: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110943 140191300101952 af.py:224] 2022-01-20 18:33:55.091309: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.110986 140191300101952 af.py:224] 2022-01-20 18:33:55.091314: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.111028 140191300101952 af.py:224] 2022-01-20 18:33:55.091321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111071 140191300101952 af.py:224] 2022-01-20 18:33:55.091326: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.111113 140191300101952 af.py:224] 2022-01-20 18:33:55.091335: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.111155 140191300101952 af.py:224] 2022-01-20 18:33:55.091342: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111197 140191300101952 af.py:224] 2022-01-20 18:33:55.091348: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111240 140191300101952 af.py:224] 2022-01-20 18:33:55.091353: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.111282 140191300101952 af.py:224] 2022-01-20 18:33:55.091360: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111325 140191300101952 af.py:224] 2022-01-20 18:33:55.091365: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.111366 140191300101952 af.py:224] 2022-01-20 18:33:55.091373: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.111408 140191300101952 af.py:224] 2022-01-20 18:33:55.091381: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111451 140191300101952 af.py:224] 2022-01-20 18:33:55.091387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111493 140191300101952 af.py:224] 2022-01-20 18:33:55.091392: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.111535 140191300101952 af.py:224] 2022-01-20 18:33:55.091401: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111577 140191300101952 af.py:224] 2022-01-20 18:33:55.091407: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.111622 140191300101952 af.py:224] 2022-01-20 18:33:55.091416: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.111665 140191300101952 af.py:224] 2022-01-20 18:33:55.091423: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111707 140191300101952 af.py:224] 2022-01-20 18:33:55.091429: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111749 140191300101952 af.py:224] 2022-01-20 18:33:55.091435: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.111791 140191300101952 af.py:224] 2022-01-20 18:33:55.091445: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111834 140191300101952 af.py:224] 2022-01-20 18:33:55.091466: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.111876 140191300101952 af.py:224] 2022-01-20 18:33:55.091477: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.111917 140191300101952 af.py:224] 2022-01-20 18:33:55.091487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.111960 140191300101952 af.py:224] 2022-01-20 18:33:55.091494: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112002 140191300101952 af.py:224] 2022-01-20 18:33:55.091499: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.112058 140191300101952 af.py:224] 2022-01-20 18:33:55.091506: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112110 140191300101952 af.py:224] 2022-01-20 18:33:55.091511: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.112160 140191300101952 af.py:224] 2022-01-20 18:33:55.091520: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.112220 140191300101952 af.py:224] 2022-01-20 18:33:55.091527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112263 140191300101952 af.py:224] 2022-01-20 18:33:55.091534: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112306 140191300101952 af.py:224] 2022-01-20 18:33:55.091539: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.112348 140191300101952 af.py:224] 2022-01-20 18:33:55.091548: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112395 140191300101952 af.py:224] 2022-01-20 18:33:55.091554: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.112437 140191300101952 af.py:224] 2022-01-20 18:33:55.091563: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.112480 140191300101952 af.py:224] 2022-01-20 18:33:55.091572: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112522 140191300101952 af.py:224] 2022-01-20 18:33:55.091578: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112564 140191300101952 af.py:224] 2022-01-20 18:33:55.091583: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.112606 140191300101952 af.py:224] 2022-01-20 18:33:55.091590: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112648 140191300101952 af.py:224] 2022-01-20 18:33:55.091595: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.112691 140191300101952 af.py:224] 2022-01-20 18:33:55.091606: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.112733 140191300101952 af.py:224] 2022-01-20 18:33:55.091615: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112776 140191300101952 af.py:224] 2022-01-20 18:33:55.091624: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112818 140191300101952 af.py:224] 2022-01-20 18:33:55.091634: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.112860 140191300101952 af.py:224] 2022-01-20 18:33:55.091645: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.112902 140191300101952 af.py:224] 2022-01-20 18:33:55.091654: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.112944 140191300101952 af.py:224] 2022-01-20 18:33:55.091670: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.112987 140191300101952 af.py:224] 2022-01-20 18:33:55.091683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113029 140191300101952 af.py:224] 2022-01-20 18:33:55.091694: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113072 140191300101952 af.py:224] 2022-01-20 18:33:55.091703: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.113118 140191300101952 af.py:224] 2022-01-20 18:33:55.091716: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113161 140191300101952 af.py:224] 2022-01-20 18:33:55.091724: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.113204 140191300101952 af.py:224] 2022-01-20 18:33:55.091739: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.113246 140191300101952 af.py:224] 2022-01-20 18:33:55.091752: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113288 140191300101952 af.py:224] 2022-01-20 18:33:55.091765: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113330 140191300101952 af.py:224] 2022-01-20 18:33:55.091775: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.113373 140191300101952 af.py:224] 2022-01-20 18:33:55.091787: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113415 140191300101952 af.py:224] 2022-01-20 18:33:55.091797: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.113458 140191300101952 af.py:224] 2022-01-20 18:33:55.091811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.113500 140191300101952 af.py:224] 2022-01-20 18:33:55.091820: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113542 140191300101952 af.py:224] 2022-01-20 18:33:55.091826: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113584 140191300101952 af.py:224] 2022-01-20 18:33:55.091831: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.113626 140191300101952 af.py:224] 2022-01-20 18:33:55.091838: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113668 140191300101952 af.py:224] 2022-01-20 18:33:55.091844: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.113710 140191300101952 af.py:224] 2022-01-20 18:33:55.091852: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.113753 140191300101952 af.py:224] 2022-01-20 18:33:55.091859: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113795 140191300101952 af.py:224] 2022-01-20 18:33:55.091869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113840 140191300101952 af.py:224] 2022-01-20 18:33:55.091878: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.113901 140191300101952 af.py:224] 2022-01-20 18:33:55.091887: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.113946 140191300101952 af.py:224] 2022-01-20 18:33:55.091893: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.113988 140191300101952 af.py:224] 2022-01-20 18:33:55.091901: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.114031 140191300101952 af.py:224] 2022-01-20 18:33:55.091910: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114073 140191300101952 af.py:224] 2022-01-20 18:33:55.091920: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114115 140191300101952 af.py:224] 2022-01-20 18:33:55.091928: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.114157 140191300101952 af.py:224] 2022-01-20 18:33:55.091940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114200 140191300101952 af.py:224] 2022-01-20 18:33:55.091949: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.114241 140191300101952 af.py:224] 2022-01-20 18:33:55.091963: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.114284 140191300101952 af.py:224] 2022-01-20 18:33:55.091975: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114326 140191300101952 af.py:224] 2022-01-20 18:33:55.091986: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114369 140191300101952 af.py:224] 2022-01-20 18:33:55.091996: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.114411 140191300101952 af.py:224] 2022-01-20 18:33:55.092006: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114453 140191300101952 af.py:224] 2022-01-20 18:33:55.092012: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.114495 140191300101952 af.py:224] 2022-01-20 18:33:55.092022: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.114537 140191300101952 af.py:224] 2022-01-20 18:33:55.092029: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114583 140191300101952 af.py:224] 2022-01-20 18:33:55.092035: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114625 140191300101952 af.py:224] 2022-01-20 18:33:55.092041: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.114668 140191300101952 af.py:224] 2022-01-20 18:33:55.092048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114710 140191300101952 af.py:224] 2022-01-20 18:33:55.092053: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.114753 140191300101952 af.py:224] 2022-01-20 18:33:55.092062: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.114795 140191300101952 af.py:224] 2022-01-20 18:33:55.092069: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114838 140191300101952 af.py:224] 2022-01-20 18:33:55.092075: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114880 140191300101952 af.py:224] 2022-01-20 18:33:55.092080: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.114922 140191300101952 af.py:224] 2022-01-20 18:33:55.092087: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.114964 140191300101952 af.py:224] 2022-01-20 18:33:55.092092: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.115006 140191300101952 af.py:224] 2022-01-20 18:33:55.092101: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.115048 140191300101952 af.py:224] 2022-01-20 18:33:55.092109: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115090 140191300101952 af.py:224] 2022-01-20 18:33:55.092120: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115133 140191300101952 af.py:224] 2022-01-20 18:33:55.092127: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.115176 140191300101952 af.py:224] 2022-01-20 18:33:55.092134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115218 140191300101952 af.py:224] 2022-01-20 18:33:55.092139: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.115263 140191300101952 af.py:224] 2022-01-20 18:33:55.092148: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.115306 140191300101952 af.py:224] 2022-01-20 18:33:55.092156: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115348 140191300101952 af.py:224] 2022-01-20 18:33:55.092162: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115391 140191300101952 af.py:224] 2022-01-20 18:33:55.092167: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.115433 140191300101952 af.py:224] 2022-01-20 18:33:55.092174: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115476 140191300101952 af.py:224] 2022-01-20 18:33:55.092179: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.115519 140191300101952 af.py:224] 2022-01-20 18:33:55.092209: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.115562 140191300101952 af.py:224] 2022-01-20 18:33:55.092218: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115621 140191300101952 af.py:224] 2022-01-20 18:33:55.092226: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115665 140191300101952 af.py:224] 2022-01-20 18:33:55.092231: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.115707 140191300101952 af.py:224] 2022-01-20 18:33:55.092237: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115749 140191300101952 af.py:224] 2022-01-20 18:33:55.092242: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.115792 140191300101952 af.py:224] 2022-01-20 18:33:55.092254: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.115834 140191300101952 af.py:224] 2022-01-20 18:33:55.092264: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115876 140191300101952 af.py:224] 2022-01-20 18:33:55.092271: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.115919 140191300101952 af.py:224] 2022-01-20 18:33:55.092276: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.115962 140191300101952 af.py:224] 2022-01-20 18:33:55.092283: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116008 140191300101952 af.py:224] 2022-01-20 18:33:55.092289: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.116051 140191300101952 af.py:224] 2022-01-20 18:33:55.092297: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.116093 140191300101952 af.py:224] 2022-01-20 18:33:55.092304: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116140 140191300101952 af.py:224] 2022-01-20 18:33:55.092311: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116183 140191300101952 af.py:224] 2022-01-20 18:33:55.092317: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.116225 140191300101952 af.py:224] 2022-01-20 18:33:55.092328: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116267 140191300101952 af.py:224] 2022-01-20 18:33:55.092338: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.116309 140191300101952 af.py:224] 2022-01-20 18:33:55.092354: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.116352 140191300101952 af.py:224] 2022-01-20 18:33:55.092366: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116394 140191300101952 af.py:224] 2022-01-20 18:33:55.092376: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116437 140191300101952 af.py:224] 2022-01-20 18:33:55.092385: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.116479 140191300101952 af.py:224] 2022-01-20 18:33:55.092396: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116521 140191300101952 af.py:224] 2022-01-20 18:33:55.092402: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.116563 140191300101952 af.py:224] 2022-01-20 18:33:55.092411: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.116606 140191300101952 af.py:224] 2022-01-20 18:33:55.092418: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116648 140191300101952 af.py:224] 2022-01-20 18:33:55.092425: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116689 140191300101952 af.py:224] 2022-01-20 18:33:55.092431: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.116734 140191300101952 af.py:224] 2022-01-20 18:33:55.092437: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116777 140191300101952 af.py:224] 2022-01-20 18:33:55.092442: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.116819 140191300101952 af.py:224] 2022-01-20 18:33:55.092452: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.116861 140191300101952 af.py:224] 2022-01-20 18:33:55.092459: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116903 140191300101952 af.py:224] 2022-01-20 18:33:55.092465: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.116945 140191300101952 af.py:224] 2022-01-20 18:33:55.092470: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.116987 140191300101952 af.py:224] 2022-01-20 18:33:55.092477: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117029 140191300101952 af.py:224] 2022-01-20 18:33:55.092482: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.117070 140191300101952 af.py:224] 2022-01-20 18:33:55.092491: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.117112 140191300101952 af.py:224] 2022-01-20 18:33:55.092498: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117155 140191300101952 af.py:224] 2022-01-20 18:33:55.092504: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117198 140191300101952 af.py:224] 2022-01-20 18:33:55.092509: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.117241 140191300101952 af.py:224] 2022-01-20 18:33:55.092515: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117298 140191300101952 af.py:224] 2022-01-20 18:33:55.092520: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.117341 140191300101952 af.py:224] 2022-01-20 18:33:55.092529: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.117383 140191300101952 af.py:224] 2022-01-20 18:33:55.092536: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117426 140191300101952 af.py:224] 2022-01-20 18:33:55.092542: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117473 140191300101952 af.py:224] 2022-01-20 18:33:55.092549: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.117515 140191300101952 af.py:224] 2022-01-20 18:33:55.092559: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117558 140191300101952 af.py:224] 2022-01-20 18:33:55.092567: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.117603 140191300101952 af.py:224] 2022-01-20 18:33:55.092581: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.117652 140191300101952 af.py:224] 2022-01-20 18:33:55.092595: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117697 140191300101952 af.py:224] 2022-01-20 18:33:55.092606: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117738 140191300101952 af.py:224] 2022-01-20 18:33:55.092614: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.117780 140191300101952 af.py:224] 2022-01-20 18:33:55.092626: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117823 140191300101952 af.py:224] 2022-01-20 18:33:55.092635: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.117864 140191300101952 af.py:224] 2022-01-20 18:33:55.092652: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.117906 140191300101952 af.py:224] 2022-01-20 18:33:55.092666: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117949 140191300101952 af.py:224] 2022-01-20 18:33:55.092677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.117992 140191300101952 af.py:224] 2022-01-20 18:33:55.092686: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.118035 140191300101952 af.py:224] 2022-01-20 18:33:55.092699: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118077 140191300101952 af.py:224] 2022-01-20 18:33:55.092708: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.118119 140191300101952 af.py:224] 2022-01-20 18:33:55.092721: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.118160 140191300101952 af.py:224] 2022-01-20 18:33:55.092730: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118205 140191300101952 af.py:224] 2022-01-20 18:33:55.092737: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118247 140191300101952 af.py:224] 2022-01-20 18:33:55.092742: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.118289 140191300101952 af.py:224] 2022-01-20 18:33:55.092749: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118330 140191300101952 af.py:224] 2022-01-20 18:33:55.092758: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.118371 140191300101952 af.py:224] 2022-01-20 18:33:55.092772: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.118412 140191300101952 af.py:224] 2022-01-20 18:33:55.092784: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118453 140191300101952 af.py:224] 2022-01-20 18:33:55.092794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118495 140191300101952 af.py:224] 2022-01-20 18:33:55.092800: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.118536 140191300101952 af.py:224] 2022-01-20 18:33:55.092808: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118577 140191300101952 af.py:224] 2022-01-20 18:33:55.092813: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.118619 140191300101952 af.py:224] 2022-01-20 18:33:55.092822: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.118660 140191300101952 af.py:224] 2022-01-20 18:33:55.092830: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118702 140191300101952 af.py:224] 2022-01-20 18:33:55.092836: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118744 140191300101952 af.py:224] 2022-01-20 18:33:55.092842: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.118786 140191300101952 af.py:224] 2022-01-20 18:33:55.092852: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118827 140191300101952 af.py:224] 2022-01-20 18:33:55.092860: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.118871 140191300101952 af.py:224] 2022-01-20 18:33:55.092874: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.118913 140191300101952 af.py:224] 2022-01-20 18:33:55.092887: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.118969 140191300101952 af.py:224] 2022-01-20 18:33:55.092897: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119012 140191300101952 af.py:224] 2022-01-20 18:33:55.092906: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.119053 140191300101952 af.py:224] 2022-01-20 18:33:55.092919: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119094 140191300101952 af.py:224] 2022-01-20 18:33:55.092928: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.119157 140191300101952 af.py:224] 2022-01-20 18:33:55.092944: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.119206 140191300101952 af.py:224] 2022-01-20 18:33:55.092956: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119249 140191300101952 af.py:224] 2022-01-20 18:33:55.092964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119291 140191300101952 af.py:224] 2022-01-20 18:33:55.092969: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.119333 140191300101952 af.py:224] 2022-01-20 18:33:55.092976: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119374 140191300101952 af.py:224] 2022-01-20 18:33:55.092981: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.119415 140191300101952 af.py:224] 2022-01-20 18:33:55.092990: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.119457 140191300101952 af.py:224] 2022-01-20 18:33:55.092997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119499 140191300101952 af.py:224] 2022-01-20 18:33:55.093005: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119540 140191300101952 af.py:224] 2022-01-20 18:33:55.093011: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.119582 140191300101952 af.py:224] 2022-01-20 18:33:55.093019: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119628 140191300101952 af.py:224] 2022-01-20 18:33:55.093024: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.119671 140191300101952 af.py:224] 2022-01-20 18:33:55.093034: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.119713 140191300101952 af.py:224] 2022-01-20 18:33:55.093042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119755 140191300101952 af.py:224] 2022-01-20 18:33:55.093048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119796 140191300101952 af.py:224] 2022-01-20 18:33:55.093053: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.119837 140191300101952 af.py:224] 2022-01-20 18:33:55.093062: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.119879 140191300101952 af.py:224] 2022-01-20 18:33:55.093069: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.119920 140191300101952 af.py:224] 2022-01-20 18:33:55.093083: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.119962 140191300101952 af.py:224] 2022-01-20 18:33:55.093096: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120004 140191300101952 af.py:224] 2022-01-20 18:33:55.093107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120046 140191300101952 af.py:224] 2022-01-20 18:33:55.093116: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.120088 140191300101952 af.py:224] 2022-01-20 18:33:55.093128: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120135 140191300101952 af.py:224] 2022-01-20 18:33:55.093136: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.120178 140191300101952 af.py:224] 2022-01-20 18:33:55.093151: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.120219 140191300101952 af.py:224] 2022-01-20 18:33:55.093164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120261 140191300101952 af.py:224] 2022-01-20 18:33:55.093175: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120302 140191300101952 af.py:224] 2022-01-20 18:33:55.093184: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.120347 140191300101952 af.py:224] 2022-01-20 18:33:55.093196: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120389 140191300101952 af.py:224] 2022-01-20 18:33:55.093206: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.120431 140191300101952 af.py:224] 2022-01-20 18:33:55.093221: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.120472 140191300101952 af.py:224] 2022-01-20 18:33:55.093235: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120514 140191300101952 af.py:224] 2022-01-20 18:33:55.093245: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120556 140191300101952 af.py:224] 2022-01-20 18:33:55.093254: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.120598 140191300101952 af.py:224] 2022-01-20 18:33:55.093266: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120657 140191300101952 af.py:224] 2022-01-20 18:33:55.093275: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.120700 140191300101952 af.py:224] 2022-01-20 18:33:55.093290: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.120742 140191300101952 af.py:224] 2022-01-20 18:33:55.093303: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120783 140191300101952 af.py:224] 2022-01-20 18:33:55.093313: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120825 140191300101952 af.py:224] 2022-01-20 18:33:55.093322: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.120867 140191300101952 af.py:224] 2022-01-20 18:33:55.093335: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.120908 140191300101952 af.py:224] 2022-01-20 18:33:55.093345: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.120950 140191300101952 af.py:224] 2022-01-20 18:33:55.093360: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.120991 140191300101952 af.py:224] 2022-01-20 18:33:55.093373: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121033 140191300101952 af.py:224] 2022-01-20 18:33:55.093383: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121080 140191300101952 af.py:224] 2022-01-20 18:33:55.093393: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.121122 140191300101952 af.py:224] 2022-01-20 18:33:55.093405: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121163 140191300101952 af.py:224] 2022-01-20 18:33:55.093414: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.121204 140191300101952 af.py:224] 2022-01-20 18:33:55.093425: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.121245 140191300101952 af.py:224] 2022-01-20 18:33:55.093437: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121287 140191300101952 af.py:224] 2022-01-20 18:33:55.093446: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121330 140191300101952 af.py:224] 2022-01-20 18:33:55.093455: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.121371 140191300101952 af.py:224] 2022-01-20 18:33:55.093467: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121414 140191300101952 af.py:224] 2022-01-20 18:33:55.093477: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.121466 140191300101952 af.py:224] 2022-01-20 18:33:55.093492: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.121524 140191300101952 af.py:224] 2022-01-20 18:33:55.093503: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121582 140191300101952 af.py:224] 2022-01-20 18:33:55.093509: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121639 140191300101952 af.py:224] 2022-01-20 18:33:55.093515: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.121696 140191300101952 af.py:224] 2022-01-20 18:33:55.093522: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121752 140191300101952 af.py:224] 2022-01-20 18:33:55.093527: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.121808 140191300101952 af.py:224] 2022-01-20 18:33:55.093535: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.121865 140191300101952 af.py:224] 2022-01-20 18:33:55.093542: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121928 140191300101952 af.py:224] 2022-01-20 18:33:55.093551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.121983 140191300101952 af.py:224] 2022-01-20 18:33:55.093558: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.122037 140191300101952 af.py:224] 2022-01-20 18:33:55.093567: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122092 140191300101952 af.py:224] 2022-01-20 18:33:55.093572: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.122148 140191300101952 af.py:224] 2022-01-20 18:33:55.093581: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.122204 140191300101952 af.py:224] 2022-01-20 18:33:55.093589: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122262 140191300101952 af.py:224] 2022-01-20 18:33:55.093596: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122308 140191300101952 af.py:224] 2022-01-20 18:33:55.093601: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.122349 140191300101952 af.py:224] 2022-01-20 18:33:55.093607: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122391 140191300101952 af.py:224] 2022-01-20 18:33:55.093613: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.122432 140191300101952 af.py:224] 2022-01-20 18:33:55.093626: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.122474 140191300101952 af.py:224] 2022-01-20 18:33:55.093639: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122535 140191300101952 af.py:224] 2022-01-20 18:33:55.093649: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122578 140191300101952 af.py:224] 2022-01-20 18:33:55.093658: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.122619 140191300101952 af.py:224] 2022-01-20 18:33:55.093670: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122661 140191300101952 af.py:224] 2022-01-20 18:33:55.093678: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.122706 140191300101952 af.py:224] 2022-01-20 18:33:55.093691: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.122748 140191300101952 af.py:224] 2022-01-20 18:33:55.093704: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122790 140191300101952 af.py:224] 2022-01-20 18:33:55.093714: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122831 140191300101952 af.py:224] 2022-01-20 18:33:55.093723: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.122873 140191300101952 af.py:224] 2022-01-20 18:33:55.093736: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.122914 140191300101952 af.py:224] 2022-01-20 18:33:55.093744: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.122955 140191300101952 af.py:224] 2022-01-20 18:33:55.093778: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.122996 140191300101952 af.py:224] 2022-01-20 18:33:55.093792: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123038 140191300101952 af.py:224] 2022-01-20 18:33:55.093802: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123079 140191300101952 af.py:224] 2022-01-20 18:33:55.093810: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.123120 140191300101952 af.py:224] 2022-01-20 18:33:55.093823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123161 140191300101952 af.py:224] 2022-01-20 18:33:55.093833: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.123202 140191300101952 af.py:224] 2022-01-20 18:33:55.093847: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.123243 140191300101952 af.py:224] 2022-01-20 18:33:55.093859: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123285 140191300101952 af.py:224] 2022-01-20 18:33:55.093869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123326 140191300101952 af.py:224] 2022-01-20 18:33:55.093877: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.123367 140191300101952 af.py:224] 2022-01-20 18:33:55.093889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123412 140191300101952 af.py:224] 2022-01-20 18:33:55.093898: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.123453 140191300101952 af.py:224] 2022-01-20 18:33:55.093914: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.123495 140191300101952 af.py:224] 2022-01-20 18:33:55.093926: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123536 140191300101952 af.py:224] 2022-01-20 18:33:55.093936: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123577 140191300101952 af.py:224] 2022-01-20 18:33:55.093946: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.123618 140191300101952 af.py:224] 2022-01-20 18:33:55.093957: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123660 140191300101952 af.py:224] 2022-01-20 18:33:55.093962: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.123701 140191300101952 af.py:224] 2022-01-20 18:33:55.093973: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.123742 140191300101952 af.py:224] 2022-01-20 18:33:55.093982: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123783 140191300101952 af.py:224] 2022-01-20 18:33:55.093989: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123824 140191300101952 af.py:224] 2022-01-20 18:33:55.093994: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.123865 140191300101952 af.py:224] 2022-01-20 18:33:55.094001: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.123907 140191300101952 af.py:224] 2022-01-20 18:33:55.094006: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.123949 140191300101952 af.py:224] 2022-01-20 18:33:55.094015: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.123990 140191300101952 af.py:224] 2022-01-20 18:33:55.094022: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124032 140191300101952 af.py:224] 2022-01-20 18:33:55.094028: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124074 140191300101952 af.py:224] 2022-01-20 18:33:55.094033: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.124124 140191300101952 af.py:224] 2022-01-20 18:33:55.094040: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124167 140191300101952 af.py:224] 2022-01-20 18:33:55.094045: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.124225 140191300101952 af.py:224] 2022-01-20 18:33:55.094054: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.124268 140191300101952 af.py:224] 2022-01-20 18:33:55.094062: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124310 140191300101952 af.py:224] 2022-01-20 18:33:55.094068: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124351 140191300101952 af.py:224] 2022-01-20 18:33:55.094073: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.124392 140191300101952 af.py:224] 2022-01-20 18:33:55.094080: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124434 140191300101952 af.py:224] 2022-01-20 18:33:55.094086: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.124476 140191300101952 af.py:224] 2022-01-20 18:33:55.094094: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.124517 140191300101952 af.py:224] 2022-01-20 18:33:55.094101: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124559 140191300101952 af.py:224] 2022-01-20 18:33:55.094107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124600 140191300101952 af.py:224] 2022-01-20 18:33:55.094112: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.124641 140191300101952 af.py:224] 2022-01-20 18:33:55.094119: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124682 140191300101952 af.py:224] 2022-01-20 18:33:55.094124: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.124723 140191300101952 af.py:224] 2022-01-20 18:33:55.094134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.124765 140191300101952 af.py:224] 2022-01-20 18:33:55.094141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124806 140191300101952 af.py:224] 2022-01-20 18:33:55.094147: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124851 140191300101952 af.py:224] 2022-01-20 18:33:55.094152: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.124892 140191300101952 af.py:224] 2022-01-20 18:33:55.094158: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.124933 140191300101952 af.py:224] 2022-01-20 18:33:55.094163: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.124974 140191300101952 af.py:224] 2022-01-20 18:33:55.094172: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.125015 140191300101952 af.py:224] 2022-01-20 18:33:55.094179: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125057 140191300101952 af.py:224] 2022-01-20 18:33:55.094185: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125098 140191300101952 af.py:224] 2022-01-20 18:33:55.094190: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.125139 140191300101952 af.py:224] 2022-01-20 18:33:55.094197: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125180 140191300101952 af.py:224] 2022-01-20 18:33:55.094202: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.125221 140191300101952 af.py:224] 2022-01-20 18:33:55.094210: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.125262 140191300101952 af.py:224] 2022-01-20 18:33:55.094218: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125303 140191300101952 af.py:224] 2022-01-20 18:33:55.094224: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125344 140191300101952 af.py:224] 2022-01-20 18:33:55.094230: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.125386 140191300101952 af.py:224] 2022-01-20 18:33:55.094236: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125427 140191300101952 af.py:224] 2022-01-20 18:33:55.094242: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.125468 140191300101952 af.py:224] 2022-01-20 18:33:55.094251: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.125512 140191300101952 af.py:224] 2022-01-20 18:33:55.094275: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125553 140191300101952 af.py:224] 2022-01-20 18:33:55.094282: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125594 140191300101952 af.py:224] 2022-01-20 18:33:55.094287: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.125635 140191300101952 af.py:224] 2022-01-20 18:33:55.094294: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125676 140191300101952 af.py:224] 2022-01-20 18:33:55.094299: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.125716 140191300101952 af.py:224] 2022-01-20 18:33:55.094308: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.125757 140191300101952 af.py:224] 2022-01-20 18:33:55.094316: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125799 140191300101952 af.py:224] 2022-01-20 18:33:55.094322: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125856 140191300101952 af.py:224] 2022-01-20 18:33:55.094327: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.125898 140191300101952 af.py:224] 2022-01-20 18:33:55.094334: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.125940 140191300101952 af.py:224] 2022-01-20 18:33:55.094339: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.125981 140191300101952 af.py:224] 2022-01-20 18:33:55.094347: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.126023 140191300101952 af.py:224] 2022-01-20 18:33:55.094354: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126065 140191300101952 af.py:224] 2022-01-20 18:33:55.094360: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126106 140191300101952 af.py:224] 2022-01-20 18:33:55.094365: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.126147 140191300101952 af.py:224] 2022-01-20 18:33:55.094372: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126189 140191300101952 af.py:224] 2022-01-20 18:33:55.094377: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.126234 140191300101952 af.py:224] 2022-01-20 18:33:55.094386: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.126276 140191300101952 af.py:224] 2022-01-20 18:33:55.094394: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126317 140191300101952 af.py:224] 2022-01-20 18:33:55.094400: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126359 140191300101952 af.py:224] 2022-01-20 18:33:55.094405: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.126400 140191300101952 af.py:224] 2022-01-20 18:33:55.094411: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126442 140191300101952 af.py:224] 2022-01-20 18:33:55.094417: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.126483 140191300101952 af.py:224] 2022-01-20 18:33:55.094426: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.126524 140191300101952 af.py:224] 2022-01-20 18:33:55.094432: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126566 140191300101952 af.py:224] 2022-01-20 18:33:55.094439: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126607 140191300101952 af.py:224] 2022-01-20 18:33:55.094444: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.126648 140191300101952 af.py:224] 2022-01-20 18:33:55.094450: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126690 140191300101952 af.py:224] 2022-01-20 18:33:55.094455: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.126731 140191300101952 af.py:224] 2022-01-20 18:33:55.094464: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.126772 140191300101952 af.py:224] 2022-01-20 18:33:55.094471: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126814 140191300101952 af.py:224] 2022-01-20 18:33:55.094477: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126856 140191300101952 af.py:224] 2022-01-20 18:33:55.094482: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.126897 140191300101952 af.py:224] 2022-01-20 18:33:55.094489: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.126941 140191300101952 af.py:224] 2022-01-20 18:33:55.094494: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.126983 140191300101952 af.py:224] 2022-01-20 18:33:55.094502: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.127025 140191300101952 af.py:224] 2022-01-20 18:33:55.094509: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127066 140191300101952 af.py:224] 2022-01-20 18:33:55.094516: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127108 140191300101952 af.py:224] 2022-01-20 18:33:55.094520: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.127149 140191300101952 af.py:224] 2022-01-20 18:33:55.094526: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127191 140191300101952 af.py:224] 2022-01-20 18:33:55.094531: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.127232 140191300101952 af.py:224] 2022-01-20 18:33:55.094540: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.127274 140191300101952 af.py:224] 2022-01-20 18:33:55.094547: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127315 140191300101952 af.py:224] 2022-01-20 18:33:55.094567: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127357 140191300101952 af.py:224] 2022-01-20 18:33:55.094573: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.127398 140191300101952 af.py:224] 2022-01-20 18:33:55.094580: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127439 140191300101952 af.py:224] 2022-01-20 18:33:55.094585: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.127497 140191300101952 af.py:224] 2022-01-20 18:33:55.094593: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.127539 140191300101952 af.py:224] 2022-01-20 18:33:55.094601: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127581 140191300101952 af.py:224] 2022-01-20 18:33:55.094607: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127622 140191300101952 af.py:224] 2022-01-20 18:33:55.094612: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.127667 140191300101952 af.py:224] 2022-01-20 18:33:55.094618: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127708 140191300101952 af.py:224] 2022-01-20 18:33:55.094624: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.127749 140191300101952 af.py:224] 2022-01-20 18:33:55.094632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.127791 140191300101952 af.py:224] 2022-01-20 18:33:55.094639: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127832 140191300101952 af.py:224] 2022-01-20 18:33:55.094645: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127874 140191300101952 af.py:224] 2022-01-20 18:33:55.094651: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.127914 140191300101952 af.py:224] 2022-01-20 18:33:55.094657: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.127956 140191300101952 af.py:224] 2022-01-20 18:33:55.094663: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.127998 140191300101952 af.py:224] 2022-01-20 18:33:55.094671: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.128039 140191300101952 af.py:224] 2022-01-20 18:33:55.094679: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128081 140191300101952 af.py:224] 2022-01-20 18:33:55.094686: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128129 140191300101952 af.py:224] 2022-01-20 18:33:55.094691: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.128173 140191300101952 af.py:224] 2022-01-20 18:33:55.094698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128215 140191300101952 af.py:224] 2022-01-20 18:33:55.094704: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.128256 140191300101952 af.py:224] 2022-01-20 18:33:55.094715: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.128297 140191300101952 af.py:224] 2022-01-20 18:33:55.094723: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128342 140191300101952 af.py:224] 2022-01-20 18:33:55.094730: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128385 140191300101952 af.py:224] 2022-01-20 18:33:55.094735: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.128426 140191300101952 af.py:224] 2022-01-20 18:33:55.094742: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128468 140191300101952 af.py:224] 2022-01-20 18:33:55.094747: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.128509 140191300101952 af.py:224] 2022-01-20 18:33:55.094756: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.128550 140191300101952 af.py:224] 2022-01-20 18:33:55.094763: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128591 140191300101952 af.py:224] 2022-01-20 18:33:55.094769: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128633 140191300101952 af.py:224] 2022-01-20 18:33:55.094774: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.128674 140191300101952 af.py:224] 2022-01-20 18:33:55.094781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128716 140191300101952 af.py:224] 2022-01-20 18:33:55.094786: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.128757 140191300101952 af.py:224] 2022-01-20 18:33:55.094795: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.128798 140191300101952 af.py:224] 2022-01-20 18:33:55.094802: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128839 140191300101952 af.py:224] 2022-01-20 18:33:55.094808: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128881 140191300101952 af.py:224] 2022-01-20 18:33:55.094813: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.128922 140191300101952 af.py:224] 2022-01-20 18:33:55.094820: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.128964 140191300101952 af.py:224] 2022-01-20 18:33:55.094825: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.129005 140191300101952 af.py:224] 2022-01-20 18:33:55.094833: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.129049 140191300101952 af.py:224] 2022-01-20 18:33:55.094840: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129091 140191300101952 af.py:224] 2022-01-20 18:33:55.094846: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129153 140191300101952 af.py:224] 2022-01-20 18:33:55.094851: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.129195 140191300101952 af.py:224] 2022-01-20 18:33:55.094857: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129237 140191300101952 af.py:224] 2022-01-20 18:33:55.094863: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.129278 140191300101952 af.py:224] 2022-01-20 18:33:55.094872: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.129320 140191300101952 af.py:224] 2022-01-20 18:33:55.094879: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129361 140191300101952 af.py:224] 2022-01-20 18:33:55.094885: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129403 140191300101952 af.py:224] 2022-01-20 18:33:55.094890: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.129444 140191300101952 af.py:224] 2022-01-20 18:33:55.094896: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129485 140191300101952 af.py:224] 2022-01-20 18:33:55.094901: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.129526 140191300101952 af.py:224] 2022-01-20 18:33:55.094910: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.129567 140191300101952 af.py:224] 2022-01-20 18:33:55.094917: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129608 140191300101952 af.py:224] 2022-01-20 18:33:55.094923: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129649 140191300101952 af.py:224] 2022-01-20 18:33:55.094928: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.129691 140191300101952 af.py:224] 2022-01-20 18:33:55.094935: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129733 140191300101952 af.py:224] 2022-01-20 18:33:55.094940: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.129778 140191300101952 af.py:224] 2022-01-20 18:33:55.094949: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.129842 140191300101952 af.py:224] 2022-01-20 18:33:55.094956: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129887 140191300101952 af.py:224] 2022-01-20 18:33:55.094962: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.129929 140191300101952 af.py:224] 2022-01-20 18:33:55.094968: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.129970 140191300101952 af.py:224] 2022-01-20 18:33:55.094975: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130011 140191300101952 af.py:224] 2022-01-20 18:33:55.094980: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.130053 140191300101952 af.py:224] 2022-01-20 18:33:55.094989: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.130108 140191300101952 af.py:224] 2022-01-20 18:33:55.094996: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130158 140191300101952 af.py:224] 2022-01-20 18:33:55.095002: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130201 140191300101952 af.py:224] 2022-01-20 18:33:55.095007: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.130243 140191300101952 af.py:224] 2022-01-20 18:33:55.095014: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130285 140191300101952 af.py:224] 2022-01-20 18:33:55.095019: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.130326 140191300101952 af.py:224] 2022-01-20 18:33:55.095028: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.130367 140191300101952 af.py:224] 2022-01-20 18:33:55.095035: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130409 140191300101952 af.py:224] 2022-01-20 18:33:55.095041: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130451 140191300101952 af.py:224] 2022-01-20 18:33:55.095046: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.130492 140191300101952 af.py:224] 2022-01-20 18:33:55.095053: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130537 140191300101952 af.py:224] 2022-01-20 18:33:55.095058: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.130579 140191300101952 af.py:224] 2022-01-20 18:33:55.095067: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.130621 140191300101952 af.py:224] 2022-01-20 18:33:55.095075: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130663 140191300101952 af.py:224] 2022-01-20 18:33:55.095081: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130704 140191300101952 af.py:224] 2022-01-20 18:33:55.095086: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.130746 140191300101952 af.py:224] 2022-01-20 18:33:55.095093: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130787 140191300101952 af.py:224] 2022-01-20 18:33:55.095098: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.130828 140191300101952 af.py:224] 2022-01-20 18:33:55.095107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.130894 140191300101952 af.py:224] 2022-01-20 18:33:55.095115: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130949 140191300101952 af.py:224] 2022-01-20 18:33:55.095122: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.130992 140191300101952 af.py:224] 2022-01-20 18:33:55.095127: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.131034 140191300101952 af.py:224] 2022-01-20 18:33:55.095135: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131074 140191300101952 af.py:224] 2022-01-20 18:33:55.095142: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.131115 140191300101952 af.py:224] 2022-01-20 18:33:55.095154: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.131157 140191300101952 af.py:224] 2022-01-20 18:33:55.095164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131198 140191300101952 af.py:224] 2022-01-20 18:33:55.095172: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131243 140191300101952 af.py:224] 2022-01-20 18:33:55.095177: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.131285 140191300101952 af.py:224] 2022-01-20 18:33:55.095184: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131326 140191300101952 af.py:224] 2022-01-20 18:33:55.095189: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.131367 140191300101952 af.py:224] 2022-01-20 18:33:55.095198: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.131409 140191300101952 af.py:224] 2022-01-20 18:33:55.095205: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131450 140191300101952 af.py:224] 2022-01-20 18:33:55.095212: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131492 140191300101952 af.py:224] 2022-01-20 18:33:55.095217: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.131533 140191300101952 af.py:224] 2022-01-20 18:33:55.095223: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131574 140191300101952 af.py:224] 2022-01-20 18:33:55.095228: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.131616 140191300101952 af.py:224] 2022-01-20 18:33:55.095237: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.131657 140191300101952 af.py:224] 2022-01-20 18:33:55.095244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131698 140191300101952 af.py:224] 2022-01-20 18:33:55.095250: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131739 140191300101952 af.py:224] 2022-01-20 18:33:55.095257: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.131781 140191300101952 af.py:224] 2022-01-20 18:33:55.095267: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131822 140191300101952 af.py:224] 2022-01-20 18:33:55.095273: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.131864 140191300101952 af.py:224] 2022-01-20 18:33:55.095282: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.131906 140191300101952 af.py:224] 2022-01-20 18:33:55.095289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131955 140191300101952 af.py:224] 2022-01-20 18:33:55.095295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.131997 140191300101952 af.py:224] 2022-01-20 18:33:55.095301: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.132038 140191300101952 af.py:224] 2022-01-20 18:33:55.095308: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132080 140191300101952 af.py:224] 2022-01-20 18:33:55.095313: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.132128 140191300101952 af.py:224] 2022-01-20 18:33:55.095326: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.132171 140191300101952 af.py:224] 2022-01-20 18:33:55.095338: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132212 140191300101952 af.py:224] 2022-01-20 18:33:55.095349: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132254 140191300101952 af.py:224] 2022-01-20 18:33:55.095358: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.132295 140191300101952 af.py:224] 2022-01-20 18:33:55.095370: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132338 140191300101952 af.py:224] 2022-01-20 18:33:55.095379: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.132379 140191300101952 af.py:224] 2022-01-20 18:33:55.095394: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.132420 140191300101952 af.py:224] 2022-01-20 18:33:55.095405: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132462 140191300101952 af.py:224] 2022-01-20 18:33:55.095435: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132503 140191300101952 af.py:224] 2022-01-20 18:33:55.095444: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.132561 140191300101952 af.py:224] 2022-01-20 18:33:55.095456: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132604 140191300101952 af.py:224] 2022-01-20 18:33:55.095465: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.132646 140191300101952 af.py:224] 2022-01-20 18:33:55.095480: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.132693 140191300101952 af.py:224] 2022-01-20 18:33:55.095491: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132735 140191300101952 af.py:224] 2022-01-20 18:33:55.095501: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132777 140191300101952 af.py:224] 2022-01-20 18:33:55.095510: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.132818 140191300101952 af.py:224] 2022-01-20 18:33:55.095522: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132860 140191300101952 af.py:224] 2022-01-20 18:33:55.095531: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.132901 140191300101952 af.py:224] 2022-01-20 18:33:55.095546: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.132942 140191300101952 af.py:224] 2022-01-20 18:33:55.095557: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.132984 140191300101952 af.py:224] 2022-01-20 18:33:55.095567: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133026 140191300101952 af.py:224] 2022-01-20 18:33:55.095576: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.133067 140191300101952 af.py:224] 2022-01-20 18:33:55.095587: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133109 140191300101952 af.py:224] 2022-01-20 18:33:55.095597: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.133150 140191300101952 af.py:224] 2022-01-20 18:33:55.095612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.133191 140191300101952 af.py:224] 2022-01-20 18:33:55.095657: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133233 140191300101952 af.py:224] 2022-01-20 18:33:55.095670: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133274 140191300101952 af.py:224] 2022-01-20 18:33:55.095679: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.133315 140191300101952 af.py:224] 2022-01-20 18:33:55.095691: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133356 140191300101952 af.py:224] 2022-01-20 18:33:55.095700: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.133401 140191300101952 af.py:224] 2022-01-20 18:33:55.095715: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.133442 140191300101952 af.py:224] 2022-01-20 18:33:55.095729: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133485 140191300101952 af.py:224] 2022-01-20 18:33:55.095740: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133526 140191300101952 af.py:224] 2022-01-20 18:33:55.095749: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.133567 140191300101952 af.py:224] 2022-01-20 18:33:55.095761: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133609 140191300101952 af.py:224] 2022-01-20 18:33:55.095770: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.133649 140191300101952 af.py:224] 2022-01-20 18:33:55.095786: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.133691 140191300101952 af.py:224] 2022-01-20 18:33:55.095798: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133732 140191300101952 af.py:224] 2022-01-20 18:33:55.095809: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133773 140191300101952 af.py:224] 2022-01-20 18:33:55.095819: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.133814 140191300101952 af.py:224] 2022-01-20 18:33:55.095832: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133856 140191300101952 af.py:224] 2022-01-20 18:33:55.095841: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.133897 140191300101952 af.py:224] 2022-01-20 18:33:55.095857: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.133937 140191300101952 af.py:224] 2022-01-20 18:33:55.095869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.133978 140191300101952 af.py:224] 2022-01-20 18:33:55.095877: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134019 140191300101952 af.py:224] 2022-01-20 18:33:55.095882: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.134061 140191300101952 af.py:224] 2022-01-20 18:33:55.095889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134106 140191300101952 af.py:224] 2022-01-20 18:33:55.095894: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.134147 140191300101952 af.py:224] 2022-01-20 18:33:55.095903: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.134204 140191300101952 af.py:224] 2022-01-20 18:33:55.095910: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134248 140191300101952 af.py:224] 2022-01-20 18:33:55.095917: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134290 140191300101952 af.py:224] 2022-01-20 18:33:55.095926: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.134331 140191300101952 af.py:224] 2022-01-20 18:33:55.095938: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134371 140191300101952 af.py:224] 2022-01-20 18:33:55.095947: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.134412 140191300101952 af.py:224] 2022-01-20 18:33:55.095962: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.134454 140191300101952 af.py:224] 2022-01-20 18:33:55.095975: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134495 140191300101952 af.py:224] 2022-01-20 18:33:55.095986: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134537 140191300101952 af.py:224] 2022-01-20 18:33:55.095995: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.134578 140191300101952 af.py:224] 2022-01-20 18:33:55.096006: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134619 140191300101952 af.py:224] 2022-01-20 18:33:55.096016: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.134661 140191300101952 af.py:224] 2022-01-20 18:33:55.096031: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.134703 140191300101952 af.py:224] 2022-01-20 18:33:55.096044: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134744 140191300101952 af.py:224] 2022-01-20 18:33:55.096055: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134789 140191300101952 af.py:224] 2022-01-20 18:33:55.096065: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.134831 140191300101952 af.py:224] 2022-01-20 18:33:55.096078: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134872 140191300101952 af.py:224] 2022-01-20 18:33:55.096087: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.134914 140191300101952 af.py:224] 2022-01-20 18:33:55.096101: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.134955 140191300101952 af.py:224] 2022-01-20 18:33:55.096117: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.134996 140191300101952 af.py:224] 2022-01-20 18:33:55.096130: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135038 140191300101952 af.py:224] 2022-01-20 18:33:55.096140: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.135079 140191300101952 af.py:224] 2022-01-20 18:33:55.096152: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135120 140191300101952 af.py:224] 2022-01-20 18:33:55.096162: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.135161 140191300101952 af.py:224] 2022-01-20 18:33:55.096177: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.135203 140191300101952 af.py:224] 2022-01-20 18:33:55.096188: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135244 140191300101952 af.py:224] 2022-01-20 18:33:55.096199: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135286 140191300101952 af.py:224] 2022-01-20 18:33:55.096209: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.135327 140191300101952 af.py:224] 2022-01-20 18:33:55.096222: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135370 140191300101952 af.py:224] 2022-01-20 18:33:55.096231: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.135412 140191300101952 af.py:224] 2022-01-20 18:33:55.096244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.135453 140191300101952 af.py:224] 2022-01-20 18:33:55.096256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135497 140191300101952 af.py:224] 2022-01-20 18:33:55.096267: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135539 140191300101952 af.py:224] 2022-01-20 18:33:55.096275: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.135580 140191300101952 af.py:224] 2022-01-20 18:33:55.096288: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135622 140191300101952 af.py:224] 2022-01-20 18:33:55.096298: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.135663 140191300101952 af.py:224] 2022-01-20 18:33:55.096312: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.135705 140191300101952 af.py:224] 2022-01-20 18:33:55.096325: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135747 140191300101952 af.py:224] 2022-01-20 18:33:55.096334: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135789 140191300101952 af.py:224] 2022-01-20 18:33:55.096339: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.135846 140191300101952 af.py:224] 2022-01-20 18:33:55.096346: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.135890 140191300101952 af.py:224] 2022-01-20 18:33:55.096351: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.135931 140191300101952 af.py:224] 2022-01-20 18:33:55.096360: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.135972 140191300101952 af.py:224] 2022-01-20 18:33:55.096367: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136014 140191300101952 af.py:224] 2022-01-20 18:33:55.096373: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136056 140191300101952 af.py:224] 2022-01-20 18:33:55.096380: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.136097 140191300101952 af.py:224] 2022-01-20 18:33:55.096392: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136144 140191300101952 af.py:224] 2022-01-20 18:33:55.096401: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.136187 140191300101952 af.py:224] 2022-01-20 18:33:55.096417: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.136232 140191300101952 af.py:224] 2022-01-20 18:33:55.096431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136274 140191300101952 af.py:224] 2022-01-20 18:33:55.096442: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136315 140191300101952 af.py:224] 2022-01-20 18:33:55.096448: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.136356 140191300101952 af.py:224] 2022-01-20 18:33:55.096455: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136398 140191300101952 af.py:224] 2022-01-20 18:33:55.096461: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.136439 140191300101952 af.py:224] 2022-01-20 18:33:55.096470: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.136481 140191300101952 af.py:224] 2022-01-20 18:33:55.096478: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136522 140191300101952 af.py:224] 2022-01-20 18:33:55.096485: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136564 140191300101952 af.py:224] 2022-01-20 18:33:55.096494: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.136606 140191300101952 af.py:224] 2022-01-20 18:33:55.096505: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136648 140191300101952 af.py:224] 2022-01-20 18:33:55.096515: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.136690 140191300101952 af.py:224] 2022-01-20 18:33:55.096530: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.136731 140191300101952 af.py:224] 2022-01-20 18:33:55.096542: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136773 140191300101952 af.py:224] 2022-01-20 18:33:55.096553: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136815 140191300101952 af.py:224] 2022-01-20 18:33:55.096562: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.136856 140191300101952 af.py:224] 2022-01-20 18:33:55.096573: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.136898 140191300101952 af.py:224] 2022-01-20 18:33:55.096583: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.136943 140191300101952 af.py:224] 2022-01-20 18:33:55.096598: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.136985 140191300101952 af.py:224] 2022-01-20 18:33:55.096611: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137027 140191300101952 af.py:224] 2022-01-20 18:33:55.096622: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137069 140191300101952 af.py:224] 2022-01-20 18:33:55.096631: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.137110 140191300101952 af.py:224] 2022-01-20 18:33:55.096643: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137151 140191300101952 af.py:224] 2022-01-20 18:33:55.096653: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.137192 140191300101952 af.py:224] 2022-01-20 18:33:55.096669: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.137234 140191300101952 af.py:224] 2022-01-20 18:33:55.096681: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137276 140191300101952 af.py:224] 2022-01-20 18:33:55.096692: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137317 140191300101952 af.py:224] 2022-01-20 18:33:55.096702: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.137359 140191300101952 af.py:224] 2022-01-20 18:33:55.096712: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137401 140191300101952 af.py:224] 2022-01-20 18:33:55.096717: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.137442 140191300101952 af.py:224] 2022-01-20 18:33:55.096726: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.137501 140191300101952 af.py:224] 2022-01-20 18:33:55.096734: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137544 140191300101952 af.py:224] 2022-01-20 18:33:55.096740: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137585 140191300101952 af.py:224] 2022-01-20 18:33:55.096745: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.137627 140191300101952 af.py:224] 2022-01-20 18:33:55.096752: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137673 140191300101952 af.py:224] 2022-01-20 18:33:55.096759: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.137715 140191300101952 af.py:224] 2022-01-20 18:33:55.096769: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.137757 140191300101952 af.py:224] 2022-01-20 18:33:55.096777: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137798 140191300101952 af.py:224] 2022-01-20 18:33:55.096784: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137840 140191300101952 af.py:224] 2022-01-20 18:33:55.096789: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.137881 140191300101952 af.py:224] 2022-01-20 18:33:55.096796: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.137923 140191300101952 af.py:224] 2022-01-20 18:33:55.096802: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.137965 140191300101952 af.py:224] 2022-01-20 18:33:55.096810: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.138007 140191300101952 af.py:224] 2022-01-20 18:33:55.096819: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138049 140191300101952 af.py:224] 2022-01-20 18:33:55.096825: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138091 140191300101952 af.py:224] 2022-01-20 18:33:55.096830: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.138133 140191300101952 af.py:224] 2022-01-20 18:33:55.096837: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138175 140191300101952 af.py:224] 2022-01-20 18:33:55.096842: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.138216 140191300101952 af.py:224] 2022-01-20 18:33:55.096853: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.138257 140191300101952 af.py:224] 2022-01-20 18:33:55.096862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138299 140191300101952 af.py:224] 2022-01-20 18:33:55.096868: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138343 140191300101952 af.py:224] 2022-01-20 18:33:55.096874: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.138385 140191300101952 af.py:224] 2022-01-20 18:33:55.096880: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138426 140191300101952 af.py:224] 2022-01-20 18:33:55.096886: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.138468 140191300101952 af.py:224] 2022-01-20 18:33:55.096894: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.138509 140191300101952 af.py:224] 2022-01-20 18:33:55.096903: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138550 140191300101952 af.py:224] 2022-01-20 18:33:55.096912: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138592 140191300101952 af.py:224] 2022-01-20 18:33:55.096920: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.138634 140191300101952 af.py:224] 2022-01-20 18:33:55.096930: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138675 140191300101952 af.py:224] 2022-01-20 18:33:55.096937: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.138716 140191300101952 af.py:224] 2022-01-20 18:33:55.096950: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.138757 140191300101952 af.py:224] 2022-01-20 18:33:55.096959: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138799 140191300101952 af.py:224] 2022-01-20 18:33:55.096968: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138840 140191300101952 af.py:224] 2022-01-20 18:33:55.096976: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.138881 140191300101952 af.py:224] 2022-01-20 18:33:55.096986: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.138923 140191300101952 af.py:224] 2022-01-20 18:33:55.096993: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.138964 140191300101952 af.py:224] 2022-01-20 18:33:55.097004: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.139006 140191300101952 af.py:224] 2022-01-20 18:33:55.097013: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139051 140191300101952 af.py:224] 2022-01-20 18:33:55.097022: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139093 140191300101952 af.py:224] 2022-01-20 18:33:55.097030: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.139149 140191300101952 af.py:224] 2022-01-20 18:33:55.097040: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139191 140191300101952 af.py:224] 2022-01-20 18:33:55.097047: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.139234 140191300101952 af.py:224] 2022-01-20 18:33:55.097077: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.139275 140191300101952 af.py:224] 2022-01-20 18:33:55.097087: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139316 140191300101952 af.py:224] 2022-01-20 18:33:55.097097: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139358 140191300101952 af.py:224] 2022-01-20 18:33:55.097106: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.139400 140191300101952 af.py:224] 2022-01-20 18:33:55.097118: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139441 140191300101952 af.py:224] 2022-01-20 18:33:55.097126: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.139482 140191300101952 af.py:224] 2022-01-20 18:33:55.097141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.139523 140191300101952 af.py:224] 2022-01-20 18:33:55.097153: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139564 140191300101952 af.py:224] 2022-01-20 18:33:55.097164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139606 140191300101952 af.py:224] 2022-01-20 18:33:55.097173: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.139647 140191300101952 af.py:224] 2022-01-20 18:33:55.097184: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139689 140191300101952 af.py:224] 2022-01-20 18:33:55.097193: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.139730 140191300101952 af.py:224] 2022-01-20 18:33:55.097208: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.139775 140191300101952 af.py:224] 2022-01-20 18:33:55.097220: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139818 140191300101952 af.py:224] 2022-01-20 18:33:55.097231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139859 140191300101952 af.py:224] 2022-01-20 18:33:55.097239: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.139900 140191300101952 af.py:224] 2022-01-20 18:33:55.097251: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.139942 140191300101952 af.py:224] 2022-01-20 18:33:55.097259: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.139984 140191300101952 af.py:224] 2022-01-20 18:33:55.097274: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.140025 140191300101952 af.py:224] 2022-01-20 18:33:55.097285: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140066 140191300101952 af.py:224] 2022-01-20 18:33:55.097295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140107 140191300101952 af.py:224] 2022-01-20 18:33:55.097304: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.140154 140191300101952 af.py:224] 2022-01-20 18:33:55.097316: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140196 140191300101952 af.py:224] 2022-01-20 18:33:55.097325: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.140237 140191300101952 af.py:224] 2022-01-20 18:33:55.097341: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.140278 140191300101952 af.py:224] 2022-01-20 18:33:55.097354: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140320 140191300101952 af.py:224] 2022-01-20 18:33:55.097365: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140362 140191300101952 af.py:224] 2022-01-20 18:33:55.097372: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.140403 140191300101952 af.py:224] 2022-01-20 18:33:55.097383: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140444 140191300101952 af.py:224] 2022-01-20 18:33:55.097390: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.140488 140191300101952 af.py:224] 2022-01-20 18:33:55.097399: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.140530 140191300101952 af.py:224] 2022-01-20 18:33:55.097406: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140572 140191300101952 af.py:224] 2022-01-20 18:33:55.097413: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140614 140191300101952 af.py:224] 2022-01-20 18:33:55.097418: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.140655 140191300101952 af.py:224] 2022-01-20 18:33:55.097425: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140697 140191300101952 af.py:224] 2022-01-20 18:33:55.097430: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.140738 140191300101952 af.py:224] 2022-01-20 18:33:55.097439: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.140781 140191300101952 af.py:224] 2022-01-20 18:33:55.097447: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140836 140191300101952 af.py:224] 2022-01-20 18:33:55.097453: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140879 140191300101952 af.py:224] 2022-01-20 18:33:55.097458: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.140920 140191300101952 af.py:224] 2022-01-20 18:33:55.097465: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.140961 140191300101952 af.py:224] 2022-01-20 18:33:55.097470: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.141003 140191300101952 af.py:224] 2022-01-20 18:33:55.097478: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.141045 140191300101952 af.py:224] 2022-01-20 18:33:55.097486: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141086 140191300101952 af.py:224] 2022-01-20 18:33:55.097492: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141128 140191300101952 af.py:224] 2022-01-20 18:33:55.097497: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.141169 140191300101952 af.py:224] 2022-01-20 18:33:55.097504: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141215 140191300101952 af.py:224] 2022-01-20 18:33:55.097511: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.141256 140191300101952 af.py:224] 2022-01-20 18:33:55.097525: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.141298 140191300101952 af.py:224] 2022-01-20 18:33:55.097537: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141340 140191300101952 af.py:224] 2022-01-20 18:33:55.097548: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141381 140191300101952 af.py:224] 2022-01-20 18:33:55.097558: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.141422 140191300101952 af.py:224] 2022-01-20 18:33:55.097571: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141463 140191300101952 af.py:224] 2022-01-20 18:33:55.097581: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.141504 140191300101952 af.py:224] 2022-01-20 18:33:55.097597: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.141545 140191300101952 af.py:224] 2022-01-20 18:33:55.097611: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141587 140191300101952 af.py:224] 2022-01-20 18:33:55.097622: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141628 140191300101952 af.py:224] 2022-01-20 18:33:55.097631: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.141670 140191300101952 af.py:224] 2022-01-20 18:33:55.097641: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141712 140191300101952 af.py:224] 2022-01-20 18:33:55.097651: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.141753 140191300101952 af.py:224] 2022-01-20 18:33:55.097666: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.141794 140191300101952 af.py:224] 2022-01-20 18:33:55.097678: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141835 140191300101952 af.py:224] 2022-01-20 18:33:55.097689: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141880 140191300101952 af.py:224] 2022-01-20 18:33:55.097699: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.141922 140191300101952 af.py:224] 2022-01-20 18:33:55.097712: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.141963 140191300101952 af.py:224] 2022-01-20 18:33:55.097722: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.142004 140191300101952 af.py:224] 2022-01-20 18:33:55.097736: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.142045 140191300101952 af.py:224] 2022-01-20 18:33:55.097750: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142087 140191300101952 af.py:224] 2022-01-20 18:33:55.097760: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142128 140191300101952 af.py:224] 2022-01-20 18:33:55.097765: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.142169 140191300101952 af.py:224] 2022-01-20 18:33:55.097772: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142209 140191300101952 af.py:224] 2022-01-20 18:33:55.097778: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.142251 140191300101952 af.py:224] 2022-01-20 18:33:55.097786: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.142292 140191300101952 af.py:224] 2022-01-20 18:33:55.097794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142334 140191300101952 af.py:224] 2022-01-20 18:33:55.097800: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142376 140191300101952 af.py:224] 2022-01-20 18:33:55.097805: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.142417 140191300101952 af.py:224] 2022-01-20 18:33:55.097812: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142477 140191300101952 af.py:224] 2022-01-20 18:33:55.097817: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.142519 140191300101952 af.py:224] 2022-01-20 18:33:55.097825: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.142561 140191300101952 af.py:224] 2022-01-20 18:33:55.097833: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142607 140191300101952 af.py:224] 2022-01-20 18:33:55.097839: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142648 140191300101952 af.py:224] 2022-01-20 18:33:55.097844: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.142690 140191300101952 af.py:224] 2022-01-20 18:33:55.097851: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142731 140191300101952 af.py:224] 2022-01-20 18:33:55.097856: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.142772 140191300101952 af.py:224] 2022-01-20 18:33:55.097865: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.142813 140191300101952 af.py:224] 2022-01-20 18:33:55.097874: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142855 140191300101952 af.py:224] 2022-01-20 18:33:55.097881: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142897 140191300101952 af.py:224] 2022-01-20 18:33:55.097886: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.142938 140191300101952 af.py:224] 2022-01-20 18:33:55.097895: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.142980 140191300101952 af.py:224] 2022-01-20 18:33:55.097905: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.143021 140191300101952 af.py:224] 2022-01-20 18:33:55.097919: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.143062 140191300101952 af.py:224] 2022-01-20 18:33:55.097932: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143104 140191300101952 af.py:224] 2022-01-20 18:33:55.097942: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143145 140191300101952 af.py:224] 2022-01-20 18:33:55.097950: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.143186 140191300101952 af.py:224] 2022-01-20 18:33:55.097962: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143227 140191300101952 af.py:224] 2022-01-20 18:33:55.097972: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.143269 140191300101952 af.py:224] 2022-01-20 18:33:55.097985: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.143314 140191300101952 af.py:224] 2022-01-20 18:33:55.097999: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143356 140191300101952 af.py:224] 2022-01-20 18:33:55.098010: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143398 140191300101952 af.py:224] 2022-01-20 18:33:55.098020: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.143439 140191300101952 af.py:224] 2022-01-20 18:33:55.098032: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143480 140191300101952 af.py:224] 2022-01-20 18:33:55.098041: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.143521 140191300101952 af.py:224] 2022-01-20 18:33:55.098056: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.143562 140191300101952 af.py:224] 2022-01-20 18:33:55.098069: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143604 140191300101952 af.py:224] 2022-01-20 18:33:55.098080: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143645 140191300101952 af.py:224] 2022-01-20 18:33:55.098089: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.143687 140191300101952 af.py:224] 2022-01-20 18:33:55.098102: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143728 140191300101952 af.py:224] 2022-01-20 18:33:55.098112: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.143769 140191300101952 af.py:224] 2022-01-20 18:33:55.098127: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.143809 140191300101952 af.py:224] 2022-01-20 18:33:55.098139: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143851 140191300101952 af.py:224] 2022-01-20 18:33:55.098149: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143892 140191300101952 af.py:224] 2022-01-20 18:33:55.098158: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.143934 140191300101952 af.py:224] 2022-01-20 18:33:55.098170: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.143975 140191300101952 af.py:224] 2022-01-20 18:33:55.098180: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.144020 140191300101952 af.py:224] 2022-01-20 18:33:55.098195: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.144062 140191300101952 af.py:224] 2022-01-20 18:33:55.098209: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144122 140191300101952 af.py:224] 2022-01-20 18:33:55.098217: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144167 140191300101952 af.py:224] 2022-01-20 18:33:55.098222: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.144209 140191300101952 af.py:224] 2022-01-20 18:33:55.098229: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144250 140191300101952 af.py:224] 2022-01-20 18:33:55.098234: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.144292 140191300101952 af.py:224] 2022-01-20 18:33:55.098264: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.144333 140191300101952 af.py:224] 2022-01-20 18:33:55.098277: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144375 140191300101952 af.py:224] 2022-01-20 18:33:55.098286: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144416 140191300101952 af.py:224] 2022-01-20 18:33:55.098295: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.144457 140191300101952 af.py:224] 2022-01-20 18:33:55.098307: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144499 140191300101952 af.py:224] 2022-01-20 18:33:55.098317: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.144541 140191300101952 af.py:224] 2022-01-20 18:33:55.098332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.144582 140191300101952 af.py:224] 2022-01-20 18:33:55.098345: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144623 140191300101952 af.py:224] 2022-01-20 18:33:55.098357: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144664 140191300101952 af.py:224] 2022-01-20 18:33:55.098366: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.144705 140191300101952 af.py:224] 2022-01-20 18:33:55.098378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144750 140191300101952 af.py:224] 2022-01-20 18:33:55.098388: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.144792 140191300101952 af.py:224] 2022-01-20 18:33:55.098400: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.144833 140191300101952 af.py:224] 2022-01-20 18:33:55.098409: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144875 140191300101952 af.py:224] 2022-01-20 18:33:55.098415: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144916 140191300101952 af.py:224] 2022-01-20 18:33:55.098421: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.144958 140191300101952 af.py:224] 2022-01-20 18:33:55.098428: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.144999 140191300101952 af.py:224] 2022-01-20 18:33:55.098433: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.145040 140191300101952 af.py:224] 2022-01-20 18:33:55.098441: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.145081 140191300101952 af.py:224] 2022-01-20 18:33:55.098452: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145122 140191300101952 af.py:224] 2022-01-20 18:33:55.098462: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145163 140191300101952 af.py:224] 2022-01-20 18:33:55.098472: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.145204 140191300101952 af.py:224] 2022-01-20 18:33:55.098484: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145246 140191300101952 af.py:224] 2022-01-20 18:33:55.098494: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.145287 140191300101952 af.py:224] 2022-01-20 18:33:55.098509: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.145328 140191300101952 af.py:224] 2022-01-20 18:33:55.098522: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145369 140191300101952 af.py:224] 2022-01-20 18:33:55.098532: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145413 140191300101952 af.py:224] 2022-01-20 18:33:55.098538: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.145455 140191300101952 af.py:224] 2022-01-20 18:33:55.098546: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145496 140191300101952 af.py:224] 2022-01-20 18:33:55.098551: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.145537 140191300101952 af.py:224] 2022-01-20 18:33:55.098559: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.145578 140191300101952 af.py:224] 2022-01-20 18:33:55.098566: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145619 140191300101952 af.py:224] 2022-01-20 18:33:55.098573: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145662 140191300101952 af.py:224] 2022-01-20 18:33:55.098578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.145704 140191300101952 af.py:224] 2022-01-20 18:33:55.098587: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145758 140191300101952 af.py:224] 2022-01-20 18:33:55.098596: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.145801 140191300101952 af.py:224] 2022-01-20 18:33:55.098612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.145842 140191300101952 af.py:224] 2022-01-20 18:33:55.098626: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145884 140191300101952 af.py:224] 2022-01-20 18:33:55.098638: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.145925 140191300101952 af.py:224] 2022-01-20 18:33:55.098647: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.145966 140191300101952 af.py:224] 2022-01-20 18:33:55.098658: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146007 140191300101952 af.py:224] 2022-01-20 18:33:55.098667: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.146049 140191300101952 af.py:224] 2022-01-20 18:33:55.098682: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.146090 140191300101952 af.py:224] 2022-01-20 18:33:55.098693: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146135 140191300101952 af.py:224] 2022-01-20 18:33:55.098699: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146177 140191300101952 af.py:224] 2022-01-20 18:33:55.098705: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.146217 140191300101952 af.py:224] 2022-01-20 18:33:55.098712: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146258 140191300101952 af.py:224] 2022-01-20 18:33:55.098717: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.146299 140191300101952 af.py:224] 2022-01-20 18:33:55.098725: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.146340 140191300101952 af.py:224] 2022-01-20 18:33:55.098732: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146382 140191300101952 af.py:224] 2022-01-20 18:33:55.098742: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146423 140191300101952 af.py:224] 2022-01-20 18:33:55.098751: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.146465 140191300101952 af.py:224] 2022-01-20 18:33:55.098763: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146507 140191300101952 af.py:224] 2022-01-20 18:33:55.098771: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.146548 140191300101952 af.py:224] 2022-01-20 18:33:55.098785: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.146590 140191300101952 af.py:224] 2022-01-20 18:33:55.098798: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146631 140191300101952 af.py:224] 2022-01-20 18:33:55.098807: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146673 140191300101952 af.py:224] 2022-01-20 18:33:55.098816: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.146714 140191300101952 af.py:224] 2022-01-20 18:33:55.098825: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146755 140191300101952 af.py:224] 2022-01-20 18:33:55.098831: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.146797 140191300101952 af.py:224] 2022-01-20 18:33:55.098839: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.146842 140191300101952 af.py:224] 2022-01-20 18:33:55.098847: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146884 140191300101952 af.py:224] 2022-01-20 18:33:55.098854: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.146926 140191300101952 af.py:224] 2022-01-20 18:33:55.098859: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.146967 140191300101952 af.py:224] 2022-01-20 18:33:55.098866: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147009 140191300101952 af.py:224] 2022-01-20 18:33:55.098871: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.147050 140191300101952 af.py:224] 2022-01-20 18:33:55.098880: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.147091 140191300101952 af.py:224] 2022-01-20 18:33:55.098887: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147132 140191300101952 af.py:224] 2022-01-20 18:33:55.098893: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147174 140191300101952 af.py:224] 2022-01-20 18:33:55.098898: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.147216 140191300101952 af.py:224] 2022-01-20 18:33:55.098905: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147258 140191300101952 af.py:224] 2022-01-20 18:33:55.098910: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.147299 140191300101952 af.py:224] 2022-01-20 18:33:55.098918: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.147340 140191300101952 af.py:224] 2022-01-20 18:33:55.098925: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147397 140191300101952 af.py:224] 2022-01-20 18:33:55.098932: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147440 140191300101952 af.py:224] 2022-01-20 18:33:55.098938: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.147482 140191300101952 af.py:224] 2022-01-20 18:33:55.098944: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147524 140191300101952 af.py:224] 2022-01-20 18:33:55.098949: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.147569 140191300101952 af.py:224] 2022-01-20 18:33:55.098957: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.147611 140191300101952 af.py:224] 2022-01-20 18:33:55.098965: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147653 140191300101952 af.py:224] 2022-01-20 18:33:55.098971: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147695 140191300101952 af.py:224] 2022-01-20 18:33:55.098976: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.147737 140191300101952 af.py:224] 2022-01-20 18:33:55.098983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147778 140191300101952 af.py:224] 2022-01-20 18:33:55.098988: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.147819 140191300101952 af.py:224] 2022-01-20 18:33:55.098996: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.147861 140191300101952 af.py:224] 2022-01-20 18:33:55.099003: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147902 140191300101952 af.py:224] 2022-01-20 18:33:55.099010: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.147943 140191300101952 af.py:224] 2022-01-20 18:33:55.099015: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.147985 140191300101952 af.py:224] 2022-01-20 18:33:55.099022: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148026 140191300101952 af.py:224] 2022-01-20 18:33:55.099027: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.148067 140191300101952 af.py:224] 2022-01-20 18:33:55.099036: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.148109 140191300101952 af.py:224] 2022-01-20 18:33:55.099043: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148160 140191300101952 af.py:224] 2022-01-20 18:33:55.099049: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148202 140191300101952 af.py:224] 2022-01-20 18:33:55.099054: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.148244 140191300101952 af.py:224] 2022-01-20 18:33:55.099061: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148289 140191300101952 af.py:224] 2022-01-20 18:33:55.099066: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.148330 140191300101952 af.py:224] 2022-01-20 18:33:55.099075: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.148371 140191300101952 af.py:224] 2022-01-20 18:33:55.099082: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148413 140191300101952 af.py:224] 2022-01-20 18:33:55.099089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148456 140191300101952 af.py:224] 2022-01-20 18:33:55.099095: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.148497 140191300101952 af.py:224] 2022-01-20 18:33:55.099101: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148538 140191300101952 af.py:224] 2022-01-20 18:33:55.099106: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.148580 140191300101952 af.py:224] 2022-01-20 18:33:55.099115: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.148622 140191300101952 af.py:224] 2022-01-20 18:33:55.099121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148663 140191300101952 af.py:224] 2022-01-20 18:33:55.099128: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148705 140191300101952 af.py:224] 2022-01-20 18:33:55.099133: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.148746 140191300101952 af.py:224] 2022-01-20 18:33:55.099140: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148787 140191300101952 af.py:224] 2022-01-20 18:33:55.099145: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.148829 140191300101952 af.py:224] 2022-01-20 18:33:55.099154: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.148869 140191300101952 af.py:224] 2022-01-20 18:33:55.099160: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148910 140191300101952 af.py:224] 2022-01-20 18:33:55.099167: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.148954 140191300101952 af.py:224] 2022-01-20 18:33:55.099172: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.148996 140191300101952 af.py:224] 2022-01-20 18:33:55.099179: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149037 140191300101952 af.py:224] 2022-01-20 18:33:55.099184: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.149099 140191300101952 af.py:224] 2022-01-20 18:33:55.099193: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.149142 140191300101952 af.py:224] 2022-01-20 18:33:55.099199: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149184 140191300101952 af.py:224] 2022-01-20 18:33:55.099205: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149226 140191300101952 af.py:224] 2022-01-20 18:33:55.099211: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.149267 140191300101952 af.py:224] 2022-01-20 18:33:55.099217: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149308 140191300101952 af.py:224] 2022-01-20 18:33:55.099222: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.149350 140191300101952 af.py:224] 2022-01-20 18:33:55.099231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.149391 140191300101952 af.py:224] 2022-01-20 18:33:55.099237: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149433 140191300101952 af.py:224] 2022-01-20 18:33:55.099244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149474 140191300101952 af.py:224] 2022-01-20 18:33:55.099250: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.149515 140191300101952 af.py:224] 2022-01-20 18:33:55.099259: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149556 140191300101952 af.py:224] 2022-01-20 18:33:55.099265: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.149597 140191300101952 af.py:224] 2022-01-20 18:33:55.099274: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.149638 140191300101952 af.py:224] 2022-01-20 18:33:55.099281: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149683 140191300101952 af.py:224] 2022-01-20 18:33:55.099287: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149725 140191300101952 af.py:224] 2022-01-20 18:33:55.099292: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.149766 140191300101952 af.py:224] 2022-01-20 18:33:55.099299: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149807 140191300101952 af.py:224] 2022-01-20 18:33:55.099304: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.149849 140191300101952 af.py:224] 2022-01-20 18:33:55.099313: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.149889 140191300101952 af.py:224] 2022-01-20 18:33:55.099319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149931 140191300101952 af.py:224] 2022-01-20 18:33:55.099326: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.149972 140191300101952 af.py:224] 2022-01-20 18:33:55.099331: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.150014 140191300101952 af.py:224] 2022-01-20 18:33:55.099338: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150055 140191300101952 af.py:224] 2022-01-20 18:33:55.099343: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.150096 140191300101952 af.py:224] 2022-01-20 18:33:55.099351: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.150138 140191300101952 af.py:224] 2022-01-20 18:33:55.099358: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150180 140191300101952 af.py:224] 2022-01-20 18:33:55.099364: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150221 140191300101952 af.py:224] 2022-01-20 18:33:55.099370: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.150263 140191300101952 af.py:224] 2022-01-20 18:33:55.099376: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150305 140191300101952 af.py:224] 2022-01-20 18:33:55.099381: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.150346 140191300101952 af.py:224] 2022-01-20 18:33:55.099389: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.150391 140191300101952 af.py:224] 2022-01-20 18:33:55.099396: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150433 140191300101952 af.py:224] 2022-01-20 18:33:55.099402: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150474 140191300101952 af.py:224] 2022-01-20 18:33:55.099408: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.150515 140191300101952 af.py:224] 2022-01-20 18:33:55.099414: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150557 140191300101952 af.py:224] 2022-01-20 18:33:55.099419: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.150598 140191300101952 af.py:224] 2022-01-20 18:33:55.099428: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.150639 140191300101952 af.py:224] 2022-01-20 18:33:55.099435: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150681 140191300101952 af.py:224] 2022-01-20 18:33:55.099441: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150739 140191300101952 af.py:224] 2022-01-20 18:33:55.099446: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.150782 140191300101952 af.py:224] 2022-01-20 18:33:55.099453: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150824 140191300101952 af.py:224] 2022-01-20 18:33:55.099458: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.150866 140191300101952 af.py:224] 2022-01-20 18:33:55.099466: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.150906 140191300101952 af.py:224] 2022-01-20 18:33:55.099473: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150948 140191300101952 af.py:224] 2022-01-20 18:33:55.099480: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.150990 140191300101952 af.py:224] 2022-01-20 18:33:55.099485: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.151031 140191300101952 af.py:224] 2022-01-20 18:33:55.099492: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151072 140191300101952 af.py:224] 2022-01-20 18:33:55.099497: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.151118 140191300101952 af.py:224] 2022-01-20 18:33:55.099505: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.151159 140191300101952 af.py:224] 2022-01-20 18:33:55.099515: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151201 140191300101952 af.py:224] 2022-01-20 18:33:55.099523: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151242 140191300101952 af.py:224] 2022-01-20 18:33:55.099532: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.151283 140191300101952 af.py:224] 2022-01-20 18:33:55.099543: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151324 140191300101952 af.py:224] 2022-01-20 18:33:55.099552: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.151365 140191300101952 af.py:224] 2022-01-20 18:33:55.099568: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.151406 140191300101952 af.py:224] 2022-01-20 18:33:55.099580: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151447 140191300101952 af.py:224] 2022-01-20 18:33:55.099591: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151489 140191300101952 af.py:224] 2022-01-20 18:33:55.099601: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.151530 140191300101952 af.py:224] 2022-01-20 18:33:55.099613: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151571 140191300101952 af.py:224] 2022-01-20 18:33:55.099623: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.151611 140191300101952 af.py:224] 2022-01-20 18:33:55.099638: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.151652 140191300101952 af.py:224] 2022-01-20 18:33:55.099651: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151692 140191300101952 af.py:224] 2022-01-20 18:33:55.099661: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151734 140191300101952 af.py:224] 2022-01-20 18:33:55.099671: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.151777 140191300101952 af.py:224] 2022-01-20 18:33:55.099683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151819 140191300101952 af.py:224] 2022-01-20 18:33:55.099693: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.151860 140191300101952 af.py:224] 2022-01-20 18:33:55.099707: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.151901 140191300101952 af.py:224] 2022-01-20 18:33:55.099721: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151943 140191300101952 af.py:224] 2022-01-20 18:33:55.099732: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.151984 140191300101952 af.py:224] 2022-01-20 18:33:55.099741: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.152025 140191300101952 af.py:224] 2022-01-20 18:33:55.099754: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152066 140191300101952 af.py:224] 2022-01-20 18:33:55.099761: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.152107 140191300101952 af.py:224] 2022-01-20 18:33:55.099775: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.152155 140191300101952 af.py:224] 2022-01-20 18:33:55.099788: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152196 140191300101952 af.py:224] 2022-01-20 18:33:55.099799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152237 140191300101952 af.py:224] 2022-01-20 18:33:55.099809: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.152279 140191300101952 af.py:224] 2022-01-20 18:33:55.099822: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152320 140191300101952 af.py:224] 2022-01-20 18:33:55.099832: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.152376 140191300101952 af.py:224] 2022-01-20 18:33:55.099847: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.152419 140191300101952 af.py:224] 2022-01-20 18:33:55.099861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152460 140191300101952 af.py:224] 2022-01-20 18:33:55.099873: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152506 140191300101952 af.py:224] 2022-01-20 18:33:55.099882: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.152548 140191300101952 af.py:224] 2022-01-20 18:33:55.099894: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152589 140191300101952 af.py:224] 2022-01-20 18:33:55.099903: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.152631 140191300101952 af.py:224] 2022-01-20 18:33:55.099918: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.152672 140191300101952 af.py:224] 2022-01-20 18:33:55.099931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152714 140191300101952 af.py:224] 2022-01-20 18:33:55.099942: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152755 140191300101952 af.py:224] 2022-01-20 18:33:55.099952: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.152795 140191300101952 af.py:224] 2022-01-20 18:33:55.099965: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152836 140191300101952 af.py:224] 2022-01-20 18:33:55.099974: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.152878 140191300101952 af.py:224] 2022-01-20 18:33:55.099990: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.152919 140191300101952 af.py:224] 2022-01-20 18:33:55.100003: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.152960 140191300101952 af.py:224] 2022-01-20 18:33:55.100014: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153001 140191300101952 af.py:224] 2022-01-20 18:33:55.100023: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.153043 140191300101952 af.py:224] 2022-01-20 18:33:55.100035: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153084 140191300101952 af.py:224] 2022-01-20 18:33:55.100044: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.153126 140191300101952 af.py:224] 2022-01-20 18:33:55.100059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.153167 140191300101952 af.py:224] 2022-01-20 18:33:55.100071: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153212 140191300101952 af.py:224] 2022-01-20 18:33:55.100081: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153254 140191300101952 af.py:224] 2022-01-20 18:33:55.100091: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.153295 140191300101952 af.py:224] 2022-01-20 18:33:55.100103: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153337 140191300101952 af.py:224] 2022-01-20 18:33:55.100112: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.153378 140191300101952 af.py:224] 2022-01-20 18:33:55.100131: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.153419 140191300101952 af.py:224] 2022-01-20 18:33:55.100145: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153461 140191300101952 af.py:224] 2022-01-20 18:33:55.100157: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153502 140191300101952 af.py:224] 2022-01-20 18:33:55.100166: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.153543 140191300101952 af.py:224] 2022-01-20 18:33:55.100179: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153584 140191300101952 af.py:224] 2022-01-20 18:33:55.100189: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.153626 140191300101952 af.py:224] 2022-01-20 18:33:55.100205: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.153666 140191300101952 af.py:224] 2022-01-20 18:33:55.100217: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153708 140191300101952 af.py:224] 2022-01-20 18:33:55.100224: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153749 140191300101952 af.py:224] 2022-01-20 18:33:55.100229: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.153791 140191300101952 af.py:224] 2022-01-20 18:33:55.100236: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153833 140191300101952 af.py:224] 2022-01-20 18:33:55.100241: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.153874 140191300101952 af.py:224] 2022-01-20 18:33:55.100250: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.153919 140191300101952 af.py:224] 2022-01-20 18:33:55.100258: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.153961 140191300101952 af.py:224] 2022-01-20 18:33:55.100266: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154017 140191300101952 af.py:224] 2022-01-20 18:33:55.100273: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.154059 140191300101952 af.py:224] 2022-01-20 18:33:55.100284: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154101 140191300101952 af.py:224] 2022-01-20 18:33:55.100293: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.154143 140191300101952 af.py:224] 2022-01-20 18:33:55.100309: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.154185 140191300101952 af.py:224] 2022-01-20 18:33:55.100321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154227 140191300101952 af.py:224] 2022-01-20 18:33:55.100331: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154268 140191300101952 af.py:224] 2022-01-20 18:33:55.100341: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.154310 140191300101952 af.py:224] 2022-01-20 18:33:55.100351: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154351 140191300101952 af.py:224] 2022-01-20 18:33:55.100358: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.154393 140191300101952 af.py:224] 2022-01-20 18:33:55.100372: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.154435 140191300101952 af.py:224] 2022-01-20 18:33:55.100386: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154476 140191300101952 af.py:224] 2022-01-20 18:33:55.100396: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154518 140191300101952 af.py:224] 2022-01-20 18:33:55.100405: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.154559 140191300101952 af.py:224] 2022-01-20 18:33:55.100417: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154605 140191300101952 af.py:224] 2022-01-20 18:33:55.100427: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.154646 140191300101952 af.py:224] 2022-01-20 18:33:55.100442: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.154688 140191300101952 af.py:224] 2022-01-20 18:33:55.100455: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154729 140191300101952 af.py:224] 2022-01-20 18:33:55.100466: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154771 140191300101952 af.py:224] 2022-01-20 18:33:55.100475: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.154812 140191300101952 af.py:224] 2022-01-20 18:33:55.100487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154854 140191300101952 af.py:224] 2022-01-20 18:33:55.100497: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.154895 140191300101952 af.py:224] 2022-01-20 18:33:55.100510: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.154936 140191300101952 af.py:224] 2022-01-20 18:33:55.100563: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.154978 140191300101952 af.py:224] 2022-01-20 18:33:55.100575: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155019 140191300101952 af.py:224] 2022-01-20 18:33:55.100580: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.155059 140191300101952 af.py:224] 2022-01-20 18:33:55.100590: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155100 140191300101952 af.py:224] 2022-01-20 18:33:55.100599: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.155141 140191300101952 af.py:224] 2022-01-20 18:33:55.100614: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.155182 140191300101952 af.py:224] 2022-01-20 18:33:55.100626: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155224 140191300101952 af.py:224] 2022-01-20 18:33:55.100637: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155266 140191300101952 af.py:224] 2022-01-20 18:33:55.100647: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.155310 140191300101952 af.py:224] 2022-01-20 18:33:55.100659: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155352 140191300101952 af.py:224] 2022-01-20 18:33:55.100668: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.155393 140191300101952 af.py:224] 2022-01-20 18:33:55.100683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.155434 140191300101952 af.py:224] 2022-01-20 18:33:55.100695: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155475 140191300101952 af.py:224] 2022-01-20 18:33:55.100702: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155516 140191300101952 af.py:224] 2022-01-20 18:33:55.100711: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.155557 140191300101952 af.py:224] 2022-01-20 18:33:55.100719: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155599 140191300101952 af.py:224] 2022-01-20 18:33:55.100725: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.155640 140191300101952 af.py:224] 2022-01-20 18:33:55.100733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.155699 140191300101952 af.py:224] 2022-01-20 18:33:55.100741: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155742 140191300101952 af.py:224] 2022-01-20 18:33:55.100748: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155784 140191300101952 af.py:224] 2022-01-20 18:33:55.100753: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.155826 140191300101952 af.py:224] 2022-01-20 18:33:55.100759: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155867 140191300101952 af.py:224] 2022-01-20 18:33:55.100765: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.155909 140191300101952 af.py:224] 2022-01-20 18:33:55.100779: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.155950 140191300101952 af.py:224] 2022-01-20 18:33:55.100793: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.155991 140191300101952 af.py:224] 2022-01-20 18:33:55.100804: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156036 140191300101952 af.py:224] 2022-01-20 18:33:55.100814: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.156078 140191300101952 af.py:224] 2022-01-20 18:33:55.100825: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156124 140191300101952 af.py:224] 2022-01-20 18:33:55.100834: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.156167 140191300101952 af.py:224] 2022-01-20 18:33:55.100848: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.156209 140191300101952 af.py:224] 2022-01-20 18:33:55.100862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156250 140191300101952 af.py:224] 2022-01-20 18:33:55.100871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156291 140191300101952 af.py:224] 2022-01-20 18:33:55.100878: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.156332 140191300101952 af.py:224] 2022-01-20 18:33:55.100889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156374 140191300101952 af.py:224] 2022-01-20 18:33:55.100898: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.156415 140191300101952 af.py:224] 2022-01-20 18:33:55.100914: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.156456 140191300101952 af.py:224] 2022-01-20 18:33:55.100929: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156498 140191300101952 af.py:224] 2022-01-20 18:33:55.100941: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156539 140191300101952 af.py:224] 2022-01-20 18:33:55.100950: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.156580 140191300101952 af.py:224] 2022-01-20 18:33:55.100962: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156621 140191300101952 af.py:224] 2022-01-20 18:33:55.100972: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.156662 140191300101952 af.py:224] 2022-01-20 18:33:55.100985: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.156703 140191300101952 af.py:224] 2022-01-20 18:33:55.100994: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156749 140191300101952 af.py:224] 2022-01-20 18:33:55.101001: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156790 140191300101952 af.py:224] 2022-01-20 18:33:55.101006: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.156831 140191300101952 af.py:224] 2022-01-20 18:33:55.101013: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156872 140191300101952 af.py:224] 2022-01-20 18:33:55.101018: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.156914 140191300101952 af.py:224] 2022-01-20 18:33:55.101029: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.156955 140191300101952 af.py:224] 2022-01-20 18:33:55.101042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.156996 140191300101952 af.py:224] 2022-01-20 18:33:55.101052: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157038 140191300101952 af.py:224] 2022-01-20 18:33:55.101061: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.157080 140191300101952 af.py:224] 2022-01-20 18:33:55.101073: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157121 140191300101952 af.py:224] 2022-01-20 18:33:55.101082: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.157163 140191300101952 af.py:224] 2022-01-20 18:33:55.101097: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.157204 140191300101952 af.py:224] 2022-01-20 18:33:55.101108: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157246 140191300101952 af.py:224] 2022-01-20 18:33:55.101119: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157288 140191300101952 af.py:224] 2022-01-20 18:33:55.101128: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.157351 140191300101952 af.py:224] 2022-01-20 18:33:55.101140: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157394 140191300101952 af.py:224] 2022-01-20 18:33:55.101149: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.157435 140191300101952 af.py:224] 2022-01-20 18:33:55.101164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.157479 140191300101952 af.py:224] 2022-01-20 18:33:55.101178: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157521 140191300101952 af.py:224] 2022-01-20 18:33:55.101190: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157563 140191300101952 af.py:224] 2022-01-20 18:33:55.101199: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.157604 140191300101952 af.py:224] 2022-01-20 18:33:55.101211: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157645 140191300101952 af.py:224] 2022-01-20 18:33:55.101221: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.157686 140191300101952 af.py:224] 2022-01-20 18:33:55.101236: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.157728 140191300101952 af.py:224] 2022-01-20 18:33:55.101249: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157770 140191300101952 af.py:224] 2022-01-20 18:33:55.101259: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157811 140191300101952 af.py:224] 2022-01-20 18:33:55.101269: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.157851 140191300101952 af.py:224] 2022-01-20 18:33:55.101281: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.157893 140191300101952 af.py:224] 2022-01-20 18:33:55.101290: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.157934 140191300101952 af.py:224] 2022-01-20 18:33:55.101305: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.157975 140191300101952 af.py:224] 2022-01-20 18:33:55.101319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158016 140191300101952 af.py:224] 2022-01-20 18:33:55.101330: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158058 140191300101952 af.py:224] 2022-01-20 18:33:55.101339: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.158099 140191300101952 af.py:224] 2022-01-20 18:33:55.101350: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158144 140191300101952 af.py:224] 2022-01-20 18:33:55.101360: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.158186 140191300101952 af.py:224] 2022-01-20 18:33:55.101375: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.158227 140191300101952 af.py:224] 2022-01-20 18:33:55.101387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158269 140191300101952 af.py:224] 2022-01-20 18:33:55.101398: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158310 140191300101952 af.py:224] 2022-01-20 18:33:55.101407: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.158351 140191300101952 af.py:224] 2022-01-20 18:33:55.101419: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158393 140191300101952 af.py:224] 2022-01-20 18:33:55.101428: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.158434 140191300101952 af.py:224] 2022-01-20 18:33:55.101442: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.158475 140191300101952 af.py:224] 2022-01-20 18:33:55.101455: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158517 140191300101952 af.py:224] 2022-01-20 18:33:55.101466: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158558 140191300101952 af.py:224] 2022-01-20 18:33:55.101475: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.158599 140191300101952 af.py:224] 2022-01-20 18:33:55.101487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158641 140191300101952 af.py:224] 2022-01-20 18:33:55.101496: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.158682 140191300101952 af.py:224] 2022-01-20 18:33:55.101511: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.158723 140191300101952 af.py:224] 2022-01-20 18:33:55.101525: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158764 140191300101952 af.py:224] 2022-01-20 18:33:55.101535: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158805 140191300101952 af.py:224] 2022-01-20 18:33:55.101542: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.158849 140191300101952 af.py:224] 2022-01-20 18:33:55.101551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.158890 140191300101952 af.py:224] 2022-01-20 18:33:55.101557: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.158932 140191300101952 af.py:224] 2022-01-20 18:33:55.101567: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.158988 140191300101952 af.py:224] 2022-01-20 18:33:55.101577: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159032 140191300101952 af.py:224] 2022-01-20 18:33:55.101584: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159073 140191300101952 af.py:224] 2022-01-20 18:33:55.101590: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.159115 140191300101952 af.py:224] 2022-01-20 18:33:55.101598: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159157 140191300101952 af.py:224] 2022-01-20 18:33:55.101604: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.159198 140191300101952 af.py:224] 2022-01-20 18:33:55.101614: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.159240 140191300101952 af.py:224] 2022-01-20 18:33:55.101622: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159281 140191300101952 af.py:224] 2022-01-20 18:33:55.101629: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159322 140191300101952 af.py:224] 2022-01-20 18:33:55.101635: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.159363 140191300101952 af.py:224] 2022-01-20 18:33:55.101644: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159404 140191300101952 af.py:224] 2022-01-20 18:33:55.101650: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.159445 140191300101952 af.py:224] 2022-01-20 18:33:55.101660: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.159487 140191300101952 af.py:224] 2022-01-20 18:33:55.101669: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159528 140191300101952 af.py:224] 2022-01-20 18:33:55.101676: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159573 140191300101952 af.py:224] 2022-01-20 18:33:55.101682: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.159615 140191300101952 af.py:224] 2022-01-20 18:33:55.101690: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159656 140191300101952 af.py:224] 2022-01-20 18:33:55.101697: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.159697 140191300101952 af.py:224] 2022-01-20 18:33:55.101707: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.159737 140191300101952 af.py:224] 2022-01-20 18:33:55.101715: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159778 140191300101952 af.py:224] 2022-01-20 18:33:55.101722: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159819 140191300101952 af.py:224] 2022-01-20 18:33:55.101728: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.159861 140191300101952 af.py:224] 2022-01-20 18:33:55.101736: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.159901 140191300101952 af.py:224] 2022-01-20 18:33:55.101742: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.159943 140191300101952 af.py:224] 2022-01-20 18:33:55.101753: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.159985 140191300101952 af.py:224] 2022-01-20 18:33:55.101761: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160027 140191300101952 af.py:224] 2022-01-20 18:33:55.101768: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160068 140191300101952 af.py:224] 2022-01-20 18:33:55.101787: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.160109 140191300101952 af.py:224] 2022-01-20 18:33:55.101809: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160157 140191300101952 af.py:224] 2022-01-20 18:33:55.101818: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.160199 140191300101952 af.py:224] 2022-01-20 18:33:55.101832: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.160241 140191300101952 af.py:224] 2022-01-20 18:33:55.101844: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160285 140191300101952 af.py:224] 2022-01-20 18:33:55.101868: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160328 140191300101952 af.py:224] 2022-01-20 18:33:55.101877: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.160369 140191300101952 af.py:224] 2022-01-20 18:33:55.101900: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160411 140191300101952 af.py:224] 2022-01-20 18:33:55.101926: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.160452 140191300101952 af.py:224] 2022-01-20 18:33:55.101941: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.160492 140191300101952 af.py:224] 2022-01-20 18:33:55.101954: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160534 140191300101952 af.py:224] 2022-01-20 18:33:55.101964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160575 140191300101952 af.py:224] 2022-01-20 18:33:55.101973: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.160633 140191300101952 af.py:224] 2022-01-20 18:33:55.101984: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160675 140191300101952 af.py:224] 2022-01-20 18:33:55.101993: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.160717 140191300101952 af.py:224] 2022-01-20 18:33:55.102007: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.160758 140191300101952 af.py:224] 2022-01-20 18:33:55.102019: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160799 140191300101952 af.py:224] 2022-01-20 18:33:55.102029: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160840 140191300101952 af.py:224] 2022-01-20 18:33:55.102037: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.160881 140191300101952 af.py:224] 2022-01-20 18:33:55.102049: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.160923 140191300101952 af.py:224] 2022-01-20 18:33:55.102058: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.160965 140191300101952 af.py:224] 2022-01-20 18:33:55.102072: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.161010 140191300101952 af.py:224] 2022-01-20 18:33:55.102083: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161052 140191300101952 af.py:224] 2022-01-20 18:33:55.102093: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161094 140191300101952 af.py:224] 2022-01-20 18:33:55.102102: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.161135 140191300101952 af.py:224] 2022-01-20 18:33:55.102113: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161177 140191300101952 af.py:224] 2022-01-20 18:33:55.102122: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.161218 140191300101952 af.py:224] 2022-01-20 18:33:55.102134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.161260 140191300101952 af.py:224] 2022-01-20 18:33:55.102146: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161301 140191300101952 af.py:224] 2022-01-20 18:33:55.102157: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161343 140191300101952 af.py:224] 2022-01-20 18:33:55.102166: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.161385 140191300101952 af.py:224] 2022-01-20 18:33:55.102178: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161426 140191300101952 af.py:224] 2022-01-20 18:33:55.102186: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.161467 140191300101952 af.py:224] 2022-01-20 18:33:55.102200: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.161508 140191300101952 af.py:224] 2022-01-20 18:33:55.102214: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161550 140191300101952 af.py:224] 2022-01-20 18:33:55.102225: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161591 140191300101952 af.py:224] 2022-01-20 18:33:55.102235: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.161632 140191300101952 af.py:224] 2022-01-20 18:33:55.102248: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161677 140191300101952 af.py:224] 2022-01-20 18:33:55.102257: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.161719 140191300101952 af.py:224] 2022-01-20 18:33:55.102271: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.161760 140191300101952 af.py:224] 2022-01-20 18:33:55.102285: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161802 140191300101952 af.py:224] 2022-01-20 18:33:55.102297: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161843 140191300101952 af.py:224] 2022-01-20 18:33:55.102307: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.161885 140191300101952 af.py:224] 2022-01-20 18:33:55.102319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.161926 140191300101952 af.py:224] 2022-01-20 18:33:55.102327: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.161968 140191300101952 af.py:224] 2022-01-20 18:33:55.102342: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.162009 140191300101952 af.py:224] 2022-01-20 18:33:55.102356: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162050 140191300101952 af.py:224] 2022-01-20 18:33:55.102367: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162091 140191300101952 af.py:224] 2022-01-20 18:33:55.102373: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.162132 140191300101952 af.py:224] 2022-01-20 18:33:55.102384: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162174 140191300101952 af.py:224] 2022-01-20 18:33:55.102393: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.162215 140191300101952 af.py:224] 2022-01-20 18:33:55.102408: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.162277 140191300101952 af.py:224] 2022-01-20 18:33:55.102420: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162321 140191300101952 af.py:224] 2022-01-20 18:33:55.102431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162363 140191300101952 af.py:224] 2022-01-20 18:33:55.102439: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.162409 140191300101952 af.py:224] 2022-01-20 18:33:55.102451: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162450 140191300101952 af.py:224] 2022-01-20 18:33:55.102461: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.162491 140191300101952 af.py:224] 2022-01-20 18:33:55.102477: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.162532 140191300101952 af.py:224] 2022-01-20 18:33:55.102493: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162573 140191300101952 af.py:224] 2022-01-20 18:33:55.102504: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162614 140191300101952 af.py:224] 2022-01-20 18:33:55.102512: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.162655 140191300101952 af.py:224] 2022-01-20 18:33:55.102520: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162697 140191300101952 af.py:224] 2022-01-20 18:33:55.102529: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.162739 140191300101952 af.py:224] 2022-01-20 18:33:55.102544: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.162780 140191300101952 af.py:224] 2022-01-20 18:33:55.102554: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162821 140191300101952 af.py:224] 2022-01-20 18:33:55.102561: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162863 140191300101952 af.py:224] 2022-01-20 18:33:55.102566: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.162904 140191300101952 af.py:224] 2022-01-20 18:33:55.102573: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.162946 140191300101952 af.py:224] 2022-01-20 18:33:55.102578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.162986 140191300101952 af.py:224] 2022-01-20 18:33:55.102587: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.163028 140191300101952 af.py:224] 2022-01-20 18:33:55.102595: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163069 140191300101952 af.py:224] 2022-01-20 18:33:55.102604: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163114 140191300101952 af.py:224] 2022-01-20 18:33:55.102609: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.163155 140191300101952 af.py:224] 2022-01-20 18:33:55.102616: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163196 140191300101952 af.py:224] 2022-01-20 18:33:55.102621: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.163237 140191300101952 af.py:224] 2022-01-20 18:33:55.102630: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.163278 140191300101952 af.py:224] 2022-01-20 18:33:55.102637: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163320 140191300101952 af.py:224] 2022-01-20 18:33:55.102644: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163361 140191300101952 af.py:224] 2022-01-20 18:33:55.102649: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.163403 140191300101952 af.py:224] 2022-01-20 18:33:55.102659: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163444 140191300101952 af.py:224] 2022-01-20 18:33:55.102668: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.163486 140191300101952 af.py:224] 2022-01-20 18:33:55.102683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.163527 140191300101952 af.py:224] 2022-01-20 18:33:55.102696: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163568 140191300101952 af.py:224] 2022-01-20 18:33:55.102706: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163610 140191300101952 af.py:224] 2022-01-20 18:33:55.102715: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.163652 140191300101952 af.py:224] 2022-01-20 18:33:55.102726: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163693 140191300101952 af.py:224] 2022-01-20 18:33:55.102735: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.163733 140191300101952 af.py:224] 2022-01-20 18:33:55.102750: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.163774 140191300101952 af.py:224] 2022-01-20 18:33:55.102763: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163819 140191300101952 af.py:224] 2022-01-20 18:33:55.102775: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163861 140191300101952 af.py:224] 2022-01-20 18:33:55.102785: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.163920 140191300101952 af.py:224] 2022-01-20 18:33:55.102798: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.163962 140191300101952 af.py:224] 2022-01-20 18:33:55.102808: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.164003 140191300101952 af.py:224] 2022-01-20 18:33:55.102822: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.164045 140191300101952 af.py:224] 2022-01-20 18:33:55.102836: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164086 140191300101952 af.py:224] 2022-01-20 18:33:55.102847: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164134 140191300101952 af.py:224] 2022-01-20 18:33:55.102857: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.164178 140191300101952 af.py:224] 2022-01-20 18:33:55.102869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164220 140191300101952 af.py:224] 2022-01-20 18:33:55.102878: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.164261 140191300101952 af.py:224] 2022-01-20 18:33:55.102893: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.164302 140191300101952 af.py:224] 2022-01-20 18:33:55.102906: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164344 140191300101952 af.py:224] 2022-01-20 18:33:55.102918: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164385 140191300101952 af.py:224] 2022-01-20 18:33:55.102928: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.164426 140191300101952 af.py:224] 2022-01-20 18:33:55.102940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164468 140191300101952 af.py:224] 2022-01-20 18:33:55.102949: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.164509 140191300101952 af.py:224] 2022-01-20 18:33:55.102964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.164555 140191300101952 af.py:224] 2022-01-20 18:33:55.102976: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164597 140191300101952 af.py:224] 2022-01-20 18:33:55.102983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164638 140191300101952 af.py:224] 2022-01-20 18:33:55.102988: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.164679 140191300101952 af.py:224] 2022-01-20 18:33:55.102995: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164720 140191300101952 af.py:224] 2022-01-20 18:33:55.103000: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.164760 140191300101952 af.py:224] 2022-01-20 18:33:55.103009: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.164802 140191300101952 af.py:224] 2022-01-20 18:33:55.103016: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164843 140191300101952 af.py:224] 2022-01-20 18:33:55.103023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164885 140191300101952 af.py:224] 2022-01-20 18:33:55.103032: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.164926 140191300101952 af.py:224] 2022-01-20 18:33:55.103041: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.164968 140191300101952 af.py:224] 2022-01-20 18:33:55.103047: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.165009 140191300101952 af.py:224] 2022-01-20 18:33:55.103056: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.165051 140191300101952 af.py:224] 2022-01-20 18:33:55.103063: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165092 140191300101952 af.py:224] 2022-01-20 18:33:55.103069: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165134 140191300101952 af.py:224] 2022-01-20 18:33:55.103076: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.165174 140191300101952 af.py:224] 2022-01-20 18:33:55.103085: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165219 140191300101952 af.py:224] 2022-01-20 18:33:55.103092: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.165261 140191300101952 af.py:224] 2022-01-20 18:33:55.103105: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.165302 140191300101952 af.py:224] 2022-01-20 18:33:55.103118: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165343 140191300101952 af.py:224] 2022-01-20 18:33:55.103129: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165385 140191300101952 af.py:224] 2022-01-20 18:33:55.103138: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.165426 140191300101952 af.py:224] 2022-01-20 18:33:55.103150: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165468 140191300101952 af.py:224] 2022-01-20 18:33:55.103158: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.165509 140191300101952 af.py:224] 2022-01-20 18:33:55.103167: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.165550 140191300101952 af.py:224] 2022-01-20 18:33:55.103175: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165607 140191300101952 af.py:224] 2022-01-20 18:33:55.103182: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165650 140191300101952 af.py:224] 2022-01-20 18:33:55.103187: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.165691 140191300101952 af.py:224] 2022-01-20 18:33:55.103194: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165733 140191300101952 af.py:224] 2022-01-20 18:33:55.103199: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.165774 140191300101952 af.py:224] 2022-01-20 18:33:55.103209: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.165815 140191300101952 af.py:224] 2022-01-20 18:33:55.103218: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165857 140191300101952 af.py:224] 2022-01-20 18:33:55.103225: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165899 140191300101952 af.py:224] 2022-01-20 18:33:55.103231: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.165944 140191300101952 af.py:224] 2022-01-20 18:33:55.103239: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.165986 140191300101952 af.py:224] 2022-01-20 18:33:55.103245: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.166027 140191300101952 af.py:224] 2022-01-20 18:33:55.103255: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.166069 140191300101952 af.py:224] 2022-01-20 18:33:55.103265: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166110 140191300101952 af.py:224] 2022-01-20 18:33:55.103272: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166152 140191300101952 af.py:224] 2022-01-20 18:33:55.103277: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.166193 140191300101952 af.py:224] 2022-01-20 18:33:55.103284: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166234 140191300101952 af.py:224] 2022-01-20 18:33:55.103289: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.166275 140191300101952 af.py:224] 2022-01-20 18:33:55.103297: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.166316 140191300101952 af.py:224] 2022-01-20 18:33:55.103305: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166357 140191300101952 af.py:224] 2022-01-20 18:33:55.103311: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166399 140191300101952 af.py:224] 2022-01-20 18:33:55.103316: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.166440 140191300101952 af.py:224] 2022-01-20 18:33:55.103324: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166482 140191300101952 af.py:224] 2022-01-20 18:33:55.103331: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.166523 140191300101952 af.py:224] 2022-01-20 18:33:55.103344: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.166565 140191300101952 af.py:224] 2022-01-20 18:33:55.103357: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166606 140191300101952 af.py:224] 2022-01-20 18:33:55.103367: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166652 140191300101952 af.py:224] 2022-01-20 18:33:55.103376: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.166693 140191300101952 af.py:224] 2022-01-20 18:33:55.103389: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166734 140191300101952 af.py:224] 2022-01-20 18:33:55.103416: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.166776 140191300101952 af.py:224] 2022-01-20 18:33:55.103431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.166817 140191300101952 af.py:224] 2022-01-20 18:33:55.103442: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166858 140191300101952 af.py:224] 2022-01-20 18:33:55.103453: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166899 140191300101952 af.py:224] 2022-01-20 18:33:55.103462: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.166941 140191300101952 af.py:224] 2022-01-20 18:33:55.103473: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.166983 140191300101952 af.py:224] 2022-01-20 18:33:55.103483: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.167025 140191300101952 af.py:224] 2022-01-20 18:33:55.103497: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.167066 140191300101952 af.py:224] 2022-01-20 18:33:55.103510: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167108 140191300101952 af.py:224] 2022-01-20 18:33:55.103519: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167150 140191300101952 af.py:224] 2022-01-20 18:33:55.103524: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.167191 140191300101952 af.py:224] 2022-01-20 18:33:55.103531: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167250 140191300101952 af.py:224] 2022-01-20 18:33:55.103536: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.167293 140191300101952 af.py:224] 2022-01-20 18:33:55.103544: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.167334 140191300101952 af.py:224] 2022-01-20 18:33:55.103551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167379 140191300101952 af.py:224] 2022-01-20 18:33:55.103558: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167421 140191300101952 af.py:224] 2022-01-20 18:33:55.103565: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.167463 140191300101952 af.py:224] 2022-01-20 18:33:55.103576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167504 140191300101952 af.py:224] 2022-01-20 18:33:55.103585: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.167551 140191300101952 af.py:224] 2022-01-20 18:33:55.103599: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.167603 140191300101952 af.py:224] 2022-01-20 18:33:55.103613: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167659 140191300101952 af.py:224] 2022-01-20 18:33:55.103624: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167716 140191300101952 af.py:224] 2022-01-20 18:33:55.103633: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.167771 140191300101952 af.py:224] 2022-01-20 18:33:55.103641: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167815 140191300101952 af.py:224] 2022-01-20 18:33:55.103646: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.167857 140191300101952 af.py:224] 2022-01-20 18:33:55.103656: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.167900 140191300101952 af.py:224] 2022-01-20 18:33:55.103668: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167942 140191300101952 af.py:224] 2022-01-20 18:33:55.103679: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.167984 140191300101952 af.py:224] 2022-01-20 18:33:55.103687: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.168037 140191300101952 af.py:224] 2022-01-20 18:33:55.103699: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168080 140191300101952 af.py:224] 2022-01-20 18:33:55.103708: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.168134 140191300101952 af.py:224] 2022-01-20 18:33:55.103723: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.168189 140191300101952 af.py:224] 2022-01-20 18:33:55.103735: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168231 140191300101952 af.py:224] 2022-01-20 18:33:55.103747: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168273 140191300101952 af.py:224] 2022-01-20 18:33:55.103758: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.168321 140191300101952 af.py:224] 2022-01-20 18:33:55.103771: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168369 140191300101952 af.py:224] 2022-01-20 18:33:55.103780: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.168417 140191300101952 af.py:224] 2022-01-20 18:33:55.103792: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.168466 140191300101952 af.py:224] 2022-01-20 18:33:55.103801: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168518 140191300101952 af.py:224] 2022-01-20 18:33:55.103807: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168570 140191300101952 af.py:224] 2022-01-20 18:33:55.103813: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.168624 140191300101952 af.py:224] 2022-01-20 18:33:55.103819: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168667 140191300101952 af.py:224] 2022-01-20 18:33:55.103825: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.168714 140191300101952 af.py:224] 2022-01-20 18:33:55.103833: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.168770 140191300101952 af.py:224] 2022-01-20 18:33:55.103841: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168824 140191300101952 af.py:224] 2022-01-20 18:33:55.103847: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168869 140191300101952 af.py:224] 2022-01-20 18:33:55.103852: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.168911 140191300101952 af.py:224] 2022-01-20 18:33:55.103859: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.168958 140191300101952 af.py:224] 2022-01-20 18:33:55.103864: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.169001 140191300101952 af.py:224] 2022-01-20 18:33:55.103873: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.169043 140191300101952 af.py:224] 2022-01-20 18:33:55.103884: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169117 140191300101952 af.py:224] 2022-01-20 18:33:55.103891: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169168 140191300101952 af.py:224] 2022-01-20 18:33:55.103896: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.169226 140191300101952 af.py:224] 2022-01-20 18:33:55.103903: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169270 140191300101952 af.py:224] 2022-01-20 18:33:55.103908: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.169312 140191300101952 af.py:224] 2022-01-20 18:33:55.103917: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.169354 140191300101952 af.py:224] 2022-01-20 18:33:55.103924: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169398 140191300101952 af.py:224] 2022-01-20 18:33:55.103931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169440 140191300101952 af.py:224] 2022-01-20 18:33:55.103939: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.169482 140191300101952 af.py:224] 2022-01-20 18:33:55.103950: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169523 140191300101952 af.py:224] 2022-01-20 18:33:55.103959: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.169565 140191300101952 af.py:224] 2022-01-20 18:33:55.103975: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.169607 140191300101952 af.py:224] 2022-01-20 18:33:55.103987: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169648 140191300101952 af.py:224] 2022-01-20 18:33:55.103998: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169690 140191300101952 af.py:224] 2022-01-20 18:33:55.104008: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.169736 140191300101952 af.py:224] 2022-01-20 18:33:55.104021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169778 140191300101952 af.py:224] 2022-01-20 18:33:55.104028: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.169819 140191300101952 af.py:224] 2022-01-20 18:33:55.104036: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.169861 140191300101952 af.py:224] 2022-01-20 18:33:55.104044: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169902 140191300101952 af.py:224] 2022-01-20 18:33:55.104051: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.169944 140191300101952 af.py:224] 2022-01-20 18:33:55.104056: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.169985 140191300101952 af.py:224] 2022-01-20 18:33:55.104063: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170026 140191300101952 af.py:224] 2022-01-20 18:33:55.104068: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.170067 140191300101952 af.py:224] 2022-01-20 18:33:55.104080: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.170108 140191300101952 af.py:224] 2022-01-20 18:33:55.104088: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170150 140191300101952 af.py:224] 2022-01-20 18:33:55.104095: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170192 140191300101952 af.py:224] 2022-01-20 18:33:55.104100: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.170232 140191300101952 af.py:224] 2022-01-20 18:33:55.104107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170273 140191300101952 af.py:224] 2022-01-20 18:33:55.104112: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.170315 140191300101952 af.py:224] 2022-01-20 18:33:55.104129: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.170356 140191300101952 af.py:224] 2022-01-20 18:33:55.104141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170397 140191300101952 af.py:224] 2022-01-20 18:33:55.104152: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170441 140191300101952 af.py:224] 2022-01-20 18:33:55.104162: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.170482 140191300101952 af.py:224] 2022-01-20 18:33:55.104175: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170524 140191300101952 af.py:224] 2022-01-20 18:33:55.104184: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.170566 140191300101952 af.py:224] 2022-01-20 18:33:55.104199: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.170607 140191300101952 af.py:224] 2022-01-20 18:33:55.104213: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170648 140191300101952 af.py:224] 2022-01-20 18:33:55.104222: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170689 140191300101952 af.py:224] 2022-01-20 18:33:55.104231: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.170731 140191300101952 af.py:224] 2022-01-20 18:33:55.104243: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170791 140191300101952 af.py:224] 2022-01-20 18:33:55.104253: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.170833 140191300101952 af.py:224] 2022-01-20 18:33:55.104268: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.170874 140191300101952 af.py:224] 2022-01-20 18:33:55.104281: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170916 140191300101952 af.py:224] 2022-01-20 18:33:55.104289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.170958 140191300101952 af.py:224] 2022-01-20 18:33:55.104294: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.170999 140191300101952 af.py:224] 2022-01-20 18:33:55.104301: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171040 140191300101952 af.py:224] 2022-01-20 18:33:55.104306: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.171081 140191300101952 af.py:224] 2022-01-20 18:33:55.104315: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.171123 140191300101952 af.py:224] 2022-01-20 18:33:55.104322: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171168 140191300101952 af.py:224] 2022-01-20 18:33:55.104328: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171209 140191300101952 af.py:224] 2022-01-20 18:33:55.104336: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.171250 140191300101952 af.py:224] 2022-01-20 18:33:55.104347: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171292 140191300101952 af.py:224] 2022-01-20 18:33:55.104357: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.171333 140191300101952 af.py:224] 2022-01-20 18:33:55.104371: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.171374 140191300101952 af.py:224] 2022-01-20 18:33:55.104383: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171415 140191300101952 af.py:224] 2022-01-20 18:33:55.104391: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171457 140191300101952 af.py:224] 2022-01-20 18:33:55.104397: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.171498 140191300101952 af.py:224] 2022-01-20 18:33:55.104403: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171540 140191300101952 af.py:224] 2022-01-20 18:33:55.104409: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.171581 140191300101952 af.py:224] 2022-01-20 18:33:55.104417: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.171622 140191300101952 af.py:224] 2022-01-20 18:33:55.104425: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171664 140191300101952 af.py:224] 2022-01-20 18:33:55.104431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171706 140191300101952 af.py:224] 2022-01-20 18:33:55.104437: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.171747 140191300101952 af.py:224] 2022-01-20 18:33:55.104443: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171789 140191300101952 af.py:224] 2022-01-20 18:33:55.104448: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.171830 140191300101952 af.py:224] 2022-01-20 18:33:55.104456: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.171875 140191300101952 af.py:224] 2022-01-20 18:33:55.104463: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171917 140191300101952 af.py:224] 2022-01-20 18:33:55.104471: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.171958 140191300101952 af.py:224] 2022-01-20 18:33:55.104477: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.171999 140191300101952 af.py:224] 2022-01-20 18:33:55.104485: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172041 140191300101952 af.py:224] 2022-01-20 18:33:55.104492: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.172082 140191300101952 af.py:224] 2022-01-20 18:33:55.104506: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.172129 140191300101952 af.py:224] 2022-01-20 18:33:55.104518: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172173 140191300101952 af.py:224] 2022-01-20 18:33:55.104530: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172215 140191300101952 af.py:224] 2022-01-20 18:33:55.104537: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.172257 140191300101952 af.py:224] 2022-01-20 18:33:55.104545: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172299 140191300101952 af.py:224] 2022-01-20 18:33:55.104550: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.172340 140191300101952 af.py:224] 2022-01-20 18:33:55.104558: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.172381 140191300101952 af.py:224] 2022-01-20 18:33:55.104566: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172438 140191300101952 af.py:224] 2022-01-20 18:33:55.104572: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172481 140191300101952 af.py:224] 2022-01-20 18:33:55.104577: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.172522 140191300101952 af.py:224] 2022-01-20 18:33:55.104586: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172568 140191300101952 af.py:224] 2022-01-20 18:33:55.104595: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.172610 140191300101952 af.py:224] 2022-01-20 18:33:55.104610: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.172651 140191300101952 af.py:224] 2022-01-20 18:33:55.104623: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172693 140191300101952 af.py:224] 2022-01-20 18:33:55.104633: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172734 140191300101952 af.py:224] 2022-01-20 18:33:55.104643: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.172775 140191300101952 af.py:224] 2022-01-20 18:33:55.104655: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172816 140191300101952 af.py:224] 2022-01-20 18:33:55.104664: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.172857 140191300101952 af.py:224] 2022-01-20 18:33:55.104679: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.172899 140191300101952 af.py:224] 2022-01-20 18:33:55.104693: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172940 140191300101952 af.py:224] 2022-01-20 18:33:55.104704: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.172981 140191300101952 af.py:224] 2022-01-20 18:33:55.104714: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.173023 140191300101952 af.py:224] 2022-01-20 18:33:55.104725: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173065 140191300101952 af.py:224] 2022-01-20 18:33:55.104731: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.173106 140191300101952 af.py:224] 2022-01-20 18:33:55.104740: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.173147 140191300101952 af.py:224] 2022-01-20 18:33:55.104748: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173188 140191300101952 af.py:224] 2022-01-20 18:33:55.104754: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173229 140191300101952 af.py:224] 2022-01-20 18:33:55.104759: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.173274 140191300101952 af.py:224] 2022-01-20 18:33:55.104766: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173316 140191300101952 af.py:224] 2022-01-20 18:33:55.104772: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.173357 140191300101952 af.py:224] 2022-01-20 18:33:55.104786: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.173398 140191300101952 af.py:224] 2022-01-20 18:33:55.104799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173439 140191300101952 af.py:224] 2022-01-20 18:33:55.104807: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173480 140191300101952 af.py:224] 2022-01-20 18:33:55.104812: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.173520 140191300101952 af.py:224] 2022-01-20 18:33:55.104818: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173562 140191300101952 af.py:224] 2022-01-20 18:33:55.104824: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.173603 140191300101952 af.py:224] 2022-01-20 18:33:55.104832: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.173645 140191300101952 af.py:224] 2022-01-20 18:33:55.104840: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173686 140191300101952 af.py:224] 2022-01-20 18:33:55.104846: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173727 140191300101952 af.py:224] 2022-01-20 18:33:55.104851: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.173768 140191300101952 af.py:224] 2022-01-20 18:33:55.104858: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173810 140191300101952 af.py:224] 2022-01-20 18:33:55.104865: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.173851 140191300101952 af.py:224] 2022-01-20 18:33:55.104877: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.173892 140191300101952 af.py:224] 2022-01-20 18:33:55.104889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173933 140191300101952 af.py:224] 2022-01-20 18:33:55.104900: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.173977 140191300101952 af.py:224] 2022-01-20 18:33:55.104909: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.174018 140191300101952 af.py:224] 2022-01-20 18:33:55.104917: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174058 140191300101952 af.py:224] 2022-01-20 18:33:55.104923: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.174114 140191300101952 af.py:224] 2022-01-20 18:33:55.104931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.174156 140191300101952 af.py:224] 2022-01-20 18:33:55.104939: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174198 140191300101952 af.py:224] 2022-01-20 18:33:55.104946: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174240 140191300101952 af.py:224] 2022-01-20 18:33:55.104951: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.174281 140191300101952 af.py:224] 2022-01-20 18:33:55.104958: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174322 140191300101952 af.py:224] 2022-01-20 18:33:55.104966: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.174364 140191300101952 af.py:224] 2022-01-20 18:33:55.104981: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.174405 140191300101952 af.py:224] 2022-01-20 18:33:55.104995: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174447 140191300101952 af.py:224] 2022-01-20 18:33:55.105007: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174488 140191300101952 af.py:224] 2022-01-20 18:33:55.105014: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.174529 140191300101952 af.py:224] 2022-01-20 18:33:55.105022: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174570 140191300101952 af.py:224] 2022-01-20 18:33:55.105027: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.174612 140191300101952 af.py:224] 2022-01-20 18:33:55.105035: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.174653 140191300101952 af.py:224] 2022-01-20 18:33:55.105043: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174697 140191300101952 af.py:224] 2022-01-20 18:33:55.105049: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174738 140191300101952 af.py:224] 2022-01-20 18:33:55.105054: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.174780 140191300101952 af.py:224] 2022-01-20 18:33:55.105062: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174821 140191300101952 af.py:224] 2022-01-20 18:33:55.105069: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.174862 140191300101952 af.py:224] 2022-01-20 18:33:55.105082: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.174903 140191300101952 af.py:224] 2022-01-20 18:33:55.105095: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174944 140191300101952 af.py:224] 2022-01-20 18:33:55.105106: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.174985 140191300101952 af.py:224] 2022-01-20 18:33:55.105113: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.175027 140191300101952 af.py:224] 2022-01-20 18:33:55.105121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175068 140191300101952 af.py:224] 2022-01-20 18:33:55.105126: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.175110 140191300101952 af.py:224] 2022-01-20 18:33:55.105134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.175151 140191300101952 af.py:224] 2022-01-20 18:33:55.105141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175192 140191300101952 af.py:224] 2022-01-20 18:33:55.105150: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175234 140191300101952 af.py:224] 2022-01-20 18:33:55.105156: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.175274 140191300101952 af.py:224] 2022-01-20 18:33:55.105162: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175315 140191300101952 af.py:224] 2022-01-20 18:33:55.105167: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.175356 140191300101952 af.py:224] 2022-01-20 18:33:55.105176: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.175403 140191300101952 af.py:224] 2022-01-20 18:33:55.105183: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175444 140191300101952 af.py:224] 2022-01-20 18:33:55.105189: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175485 140191300101952 af.py:224] 2022-01-20 18:33:55.105194: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.175525 140191300101952 af.py:224] 2022-01-20 18:33:55.105201: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175567 140191300101952 af.py:224] 2022-01-20 18:33:55.105206: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.175608 140191300101952 af.py:224] 2022-01-20 18:33:55.105214: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.175648 140191300101952 af.py:224] 2022-01-20 18:33:55.105221: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175690 140191300101952 af.py:224] 2022-01-20 18:33:55.105227: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175750 140191300101952 af.py:224] 2022-01-20 18:33:55.105233: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.175792 140191300101952 af.py:224] 2022-01-20 18:33:55.105239: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175834 140191300101952 af.py:224] 2022-01-20 18:33:55.105244: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.175875 140191300101952 af.py:224] 2022-01-20 18:33:55.105253: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.175916 140191300101952 af.py:224] 2022-01-20 18:33:55.105259: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.175958 140191300101952 af.py:224] 2022-01-20 18:33:55.105265: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176000 140191300101952 af.py:224] 2022-01-20 18:33:55.105271: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.176041 140191300101952 af.py:224] 2022-01-20 18:33:55.105278: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176087 140191300101952 af.py:224] 2022-01-20 18:33:55.105286: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.176134 140191300101952 af.py:224] 2022-01-20 18:33:55.105299: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.176178 140191300101952 af.py:224] 2022-01-20 18:33:55.105312: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176220 140191300101952 af.py:224] 2022-01-20 18:33:55.105323: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176261 140191300101952 af.py:224] 2022-01-20 18:33:55.105331: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.176303 140191300101952 af.py:224] 2022-01-20 18:33:55.105343: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176345 140191300101952 af.py:224] 2022-01-20 18:33:55.105353: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.176386 140191300101952 af.py:224] 2022-01-20 18:33:55.105368: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.176428 140191300101952 af.py:224] 2022-01-20 18:33:55.105381: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176469 140191300101952 af.py:224] 2022-01-20 18:33:55.105393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176510 140191300101952 af.py:224] 2022-01-20 18:33:55.105402: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.176552 140191300101952 af.py:224] 2022-01-20 18:33:55.105415: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176593 140191300101952 af.py:224] 2022-01-20 18:33:55.105424: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.176634 140191300101952 af.py:224] 2022-01-20 18:33:55.105440: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.176675 140191300101952 af.py:224] 2022-01-20 18:33:55.105454: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176717 140191300101952 af.py:224] 2022-01-20 18:33:55.105465: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176759 140191300101952 af.py:224] 2022-01-20 18:33:55.105475: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.176807 140191300101952 af.py:224] 2022-01-20 18:33:55.105487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176849 140191300101952 af.py:224] 2022-01-20 18:33:55.105497: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.176891 140191300101952 af.py:224] 2022-01-20 18:33:55.105512: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.176932 140191300101952 af.py:224] 2022-01-20 18:33:55.105527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.176974 140191300101952 af.py:224] 2022-01-20 18:33:55.105538: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177015 140191300101952 af.py:224] 2022-01-20 18:33:55.105545: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.177057 140191300101952 af.py:224] 2022-01-20 18:33:55.105557: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177098 140191300101952 af.py:224] 2022-01-20 18:33:55.105566: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.177140 140191300101952 af.py:224] 2022-01-20 18:33:55.105580: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.177186 140191300101952 af.py:224] 2022-01-20 18:33:55.105593: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177238 140191300101952 af.py:224] 2022-01-20 18:33:55.105602: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177291 140191300101952 af.py:224] 2022-01-20 18:33:55.105610: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.177334 140191300101952 af.py:224] 2022-01-20 18:33:55.105623: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177375 140191300101952 af.py:224] 2022-01-20 18:33:55.105631: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.177434 140191300101952 af.py:224] 2022-01-20 18:33:55.105645: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.177476 140191300101952 af.py:224] 2022-01-20 18:33:55.105657: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177517 140191300101952 af.py:224] 2022-01-20 18:33:55.105667: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177562 140191300101952 af.py:224] 2022-01-20 18:33:55.105676: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.177604 140191300101952 af.py:224] 2022-01-20 18:33:55.105688: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177645 140191300101952 af.py:224] 2022-01-20 18:33:55.105697: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.177686 140191300101952 af.py:224] 2022-01-20 18:33:55.105711: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.177728 140191300101952 af.py:224] 2022-01-20 18:33:55.105724: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177770 140191300101952 af.py:224] 2022-01-20 18:33:55.105734: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177811 140191300101952 af.py:224] 2022-01-20 18:33:55.105742: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.177852 140191300101952 af.py:224] 2022-01-20 18:33:55.105754: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.177893 140191300101952 af.py:224] 2022-01-20 18:33:55.105782: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.177934 140191300101952 af.py:224] 2022-01-20 18:33:55.105794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.177976 140191300101952 af.py:224] 2022-01-20 18:33:55.105805: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178017 140191300101952 af.py:224] 2022-01-20 18:33:55.105817: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178058 140191300101952 af.py:224] 2022-01-20 18:33:55.105825: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.178099 140191300101952 af.py:224] 2022-01-20 18:33:55.105837: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178140 140191300101952 af.py:224] 2022-01-20 18:33:55.105846: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.178181 140191300101952 af.py:224] 2022-01-20 18:33:55.105860: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.178222 140191300101952 af.py:224] 2022-01-20 18:33:55.105871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178266 140191300101952 af.py:224] 2022-01-20 18:33:55.105882: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178308 140191300101952 af.py:224] 2022-01-20 18:33:55.105891: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.178349 140191300101952 af.py:224] 2022-01-20 18:33:55.105902: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178391 140191300101952 af.py:224] 2022-01-20 18:33:55.105907: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.178432 140191300101952 af.py:224] 2022-01-20 18:33:55.105918: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.178474 140191300101952 af.py:224] 2022-01-20 18:33:55.105927: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178515 140191300101952 af.py:224] 2022-01-20 18:33:55.105933: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178557 140191300101952 af.py:224] 2022-01-20 18:33:55.105941: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.178597 140191300101952 af.py:224] 2022-01-20 18:33:55.105951: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178638 140191300101952 af.py:224] 2022-01-20 18:33:55.105960: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.178680 140191300101952 af.py:224] 2022-01-20 18:33:55.105974: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.178721 140191300101952 af.py:224] 2022-01-20 18:33:55.105988: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178763 140191300101952 af.py:224] 2022-01-20 18:33:55.105997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178804 140191300101952 af.py:224] 2022-01-20 18:33:55.106006: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.178845 140191300101952 af.py:224] 2022-01-20 18:33:55.106018: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.178886 140191300101952 af.py:224] 2022-01-20 18:33:55.106028: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.178930 140191300101952 af.py:224] 2022-01-20 18:33:55.106040: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.178972 140191300101952 af.py:224] 2022-01-20 18:33:55.106048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179013 140191300101952 af.py:224] 2022-01-20 18:33:55.106055: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179070 140191300101952 af.py:224] 2022-01-20 18:33:55.106060: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.179112 140191300101952 af.py:224] 2022-01-20 18:33:55.106067: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179154 140191300101952 af.py:224] 2022-01-20 18:33:55.106072: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.179195 140191300101952 af.py:224] 2022-01-20 18:33:55.106080: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.179236 140191300101952 af.py:224] 2022-01-20 18:33:55.106090: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179277 140191300101952 af.py:224] 2022-01-20 18:33:55.106100: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179319 140191300101952 af.py:224] 2022-01-20 18:33:55.106109: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.179360 140191300101952 af.py:224] 2022-01-20 18:33:55.106121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179401 140191300101952 af.py:224] 2022-01-20 18:33:55.106131: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.179443 140191300101952 af.py:224] 2022-01-20 18:33:55.106141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.179484 140191300101952 af.py:224] 2022-01-20 18:33:55.106149: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179526 140191300101952 af.py:224] 2022-01-20 18:33:55.106156: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179568 140191300101952 af.py:224] 2022-01-20 18:33:55.106161: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.179609 140191300101952 af.py:224] 2022-01-20 18:33:55.106168: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179654 140191300101952 af.py:224] 2022-01-20 18:33:55.106173: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.179696 140191300101952 af.py:224] 2022-01-20 18:33:55.106182: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.179738 140191300101952 af.py:224] 2022-01-20 18:33:55.106189: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179780 140191300101952 af.py:224] 2022-01-20 18:33:55.106195: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179821 140191300101952 af.py:224] 2022-01-20 18:33:55.106200: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.179862 140191300101952 af.py:224] 2022-01-20 18:33:55.106207: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.179904 140191300101952 af.py:224] 2022-01-20 18:33:55.106213: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.179945 140191300101952 af.py:224] 2022-01-20 18:33:55.106221: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.179986 140191300101952 af.py:224] 2022-01-20 18:33:55.106228: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180027 140191300101952 af.py:224] 2022-01-20 18:33:55.106234: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180069 140191300101952 af.py:224] 2022-01-20 18:33:55.106239: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.180111 140191300101952 af.py:224] 2022-01-20 18:33:55.106247: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180160 140191300101952 af.py:224] 2022-01-20 18:33:55.106254: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.180202 140191300101952 af.py:224] 2022-01-20 18:33:55.106267: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.180243 140191300101952 af.py:224] 2022-01-20 18:33:55.106280: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180285 140191300101952 af.py:224] 2022-01-20 18:33:55.106288: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180326 140191300101952 af.py:224] 2022-01-20 18:33:55.106294: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.180371 140191300101952 af.py:224] 2022-01-20 18:33:55.106301: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180413 140191300101952 af.py:224] 2022-01-20 18:33:55.106306: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.180454 140191300101952 af.py:224] 2022-01-20 18:33:55.106315: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.180496 140191300101952 af.py:224] 2022-01-20 18:33:55.106322: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180538 140191300101952 af.py:224] 2022-01-20 18:33:55.106329: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180579 140191300101952 af.py:224] 2022-01-20 18:33:55.106337: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.180621 140191300101952 af.py:224] 2022-01-20 18:33:55.106349: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180663 140191300101952 af.py:224] 2022-01-20 18:33:55.106359: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.180704 140191300101952 af.py:224] 2022-01-20 18:33:55.106374: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.180761 140191300101952 af.py:224] 2022-01-20 18:33:55.106386: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180803 140191300101952 af.py:224] 2022-01-20 18:33:55.106397: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180845 140191300101952 af.py:224] 2022-01-20 18:33:55.106406: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.180886 140191300101952 af.py:224] 2022-01-20 18:33:55.106419: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.180928 140191300101952 af.py:224] 2022-01-20 18:33:55.106428: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.180969 140191300101952 af.py:224] 2022-01-20 18:33:55.106439: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.181010 140191300101952 af.py:224] 2022-01-20 18:33:55.106451: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181050 140191300101952 af.py:224] 2022-01-20 18:33:55.106462: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181095 140191300101952 af.py:224] 2022-01-20 18:33:55.106471: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.181136 140191300101952 af.py:224] 2022-01-20 18:33:55.106482: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181179 140191300101952 af.py:224] 2022-01-20 18:33:55.106491: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.181220 140191300101952 af.py:224] 2022-01-20 18:33:55.106501: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.181262 140191300101952 af.py:224] 2022-01-20 18:33:55.106509: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181303 140191300101952 af.py:224] 2022-01-20 18:33:55.106515: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181344 140191300101952 af.py:224] 2022-01-20 18:33:55.106521: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.181385 140191300101952 af.py:224] 2022-01-20 18:33:55.106527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181427 140191300101952 af.py:224] 2022-01-20 18:33:55.106533: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.181468 140191300101952 af.py:224] 2022-01-20 18:33:55.106546: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.181509 140191300101952 af.py:224] 2022-01-20 18:33:55.106559: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181550 140191300101952 af.py:224] 2022-01-20 18:33:55.106569: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181591 140191300101952 af.py:224] 2022-01-20 18:33:55.106578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.181632 140191300101952 af.py:224] 2022-01-20 18:33:55.106589: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181674 140191300101952 af.py:224] 2022-01-20 18:33:55.106598: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.181715 140191300101952 af.py:224] 2022-01-20 18:33:55.106612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.181756 140191300101952 af.py:224] 2022-01-20 18:33:55.106625: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181801 140191300101952 af.py:224] 2022-01-20 18:33:55.106636: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181844 140191300101952 af.py:224] 2022-01-20 18:33:55.106646: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.181885 140191300101952 af.py:224] 2022-01-20 18:33:55.106659: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.181927 140191300101952 af.py:224] 2022-01-20 18:33:55.106668: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.181969 140191300101952 af.py:224] 2022-01-20 18:33:55.106684: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.182010 140191300101952 af.py:224] 2022-01-20 18:33:55.106696: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182050 140191300101952 af.py:224] 2022-01-20 18:33:55.106707: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182092 140191300101952 af.py:224] 2022-01-20 18:33:55.106716: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.182133 140191300101952 af.py:224] 2022-01-20 18:33:55.106729: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182175 140191300101952 af.py:224] 2022-01-20 18:33:55.106738: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.182216 140191300101952 af.py:224] 2022-01-20 18:33:55.106752: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.182257 140191300101952 af.py:224] 2022-01-20 18:33:55.106765: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182297 140191300101952 af.py:224] 2022-01-20 18:33:55.106777: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182338 140191300101952 af.py:224] 2022-01-20 18:33:55.106787: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.182400 140191300101952 af.py:224] 2022-01-20 18:33:55.106796: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182443 140191300101952 af.py:224] 2022-01-20 18:33:55.106802: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.182488 140191300101952 af.py:224] 2022-01-20 18:33:55.106811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.182530 140191300101952 af.py:224] 2022-01-20 18:33:55.106818: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182572 140191300101952 af.py:224] 2022-01-20 18:33:55.106825: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182613 140191300101952 af.py:224] 2022-01-20 18:33:55.106830: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.182655 140191300101952 af.py:224] 2022-01-20 18:33:55.106837: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182696 140191300101952 af.py:224] 2022-01-20 18:33:55.106844: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.182737 140191300101952 af.py:224] 2022-01-20 18:33:55.106858: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.182779 140191300101952 af.py:224] 2022-01-20 18:33:55.106870: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182820 140191300101952 af.py:224] 2022-01-20 18:33:55.106881: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182862 140191300101952 af.py:224] 2022-01-20 18:33:55.106890: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.182904 140191300101952 af.py:224] 2022-01-20 18:33:55.106902: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.182945 140191300101952 af.py:224] 2022-01-20 18:33:55.106911: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.182986 140191300101952 af.py:224] 2022-01-20 18:33:55.106923: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.183027 140191300101952 af.py:224] 2022-01-20 18:33:55.106931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183068 140191300101952 af.py:224] 2022-01-20 18:33:55.106938: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183109 140191300101952 af.py:224] 2022-01-20 18:33:55.106943: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.183151 140191300101952 af.py:224] 2022-01-20 18:33:55.106950: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183195 140191300101952 af.py:224] 2022-01-20 18:33:55.106955: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.183237 140191300101952 af.py:224] 2022-01-20 18:33:55.106965: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.183279 140191300101952 af.py:224] 2022-01-20 18:33:55.106976: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183320 140191300101952 af.py:224] 2022-01-20 18:33:55.106986: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183362 140191300101952 af.py:224] 2022-01-20 18:33:55.106996: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.183403 140191300101952 af.py:224] 2022-01-20 18:33:55.107007: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183444 140191300101952 af.py:224] 2022-01-20 18:33:55.107016: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.183486 140191300101952 af.py:224] 2022-01-20 18:33:55.107030: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.183527 140191300101952 af.py:224] 2022-01-20 18:33:55.107042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183569 140191300101952 af.py:224] 2022-01-20 18:33:55.107053: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183610 140191300101952 af.py:224] 2022-01-20 18:33:55.107061: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.183652 140191300101952 af.py:224] 2022-01-20 18:33:55.107072: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183694 140191300101952 af.py:224] 2022-01-20 18:33:55.107082: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.183735 140191300101952 af.py:224] 2022-01-20 18:33:55.107097: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.183776 140191300101952 af.py:224] 2022-01-20 18:33:55.107109: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183817 140191300101952 af.py:224] 2022-01-20 18:33:55.107120: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183859 140191300101952 af.py:224] 2022-01-20 18:33:55.107129: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.183903 140191300101952 af.py:224] 2022-01-20 18:33:55.107141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.183945 140191300101952 af.py:224] 2022-01-20 18:33:55.107150: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.183987 140191300101952 af.py:224] 2022-01-20 18:33:55.107163: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.184046 140191300101952 af.py:224] 2022-01-20 18:33:55.107174: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184089 140191300101952 af.py:224] 2022-01-20 18:33:55.107181: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184138 140191300101952 af.py:224] 2022-01-20 18:33:55.107186: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.184182 140191300101952 af.py:224] 2022-01-20 18:33:55.107193: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184223 140191300101952 af.py:224] 2022-01-20 18:33:55.107198: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.184265 140191300101952 af.py:224] 2022-01-20 18:33:55.107206: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.184307 140191300101952 af.py:224] 2022-01-20 18:33:55.107215: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184348 140191300101952 af.py:224] 2022-01-20 18:33:55.107225: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184389 140191300101952 af.py:224] 2022-01-20 18:33:55.107234: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.184430 140191300101952 af.py:224] 2022-01-20 18:33:55.107245: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184472 140191300101952 af.py:224] 2022-01-20 18:33:55.107255: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.184513 140191300101952 af.py:224] 2022-01-20 18:33:55.107269: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.184555 140191300101952 af.py:224] 2022-01-20 18:33:55.107279: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184596 140191300101952 af.py:224] 2022-01-20 18:33:55.107286: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184641 140191300101952 af.py:224] 2022-01-20 18:33:55.107291: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.184682 140191300101952 af.py:224] 2022-01-20 18:33:55.107298: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184724 140191300101952 af.py:224] 2022-01-20 18:33:55.107303: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.184764 140191300101952 af.py:224] 2022-01-20 18:33:55.107312: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.184805 140191300101952 af.py:224] 2022-01-20 18:33:55.107321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184847 140191300101952 af.py:224] 2022-01-20 18:33:55.107332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184888 140191300101952 af.py:224] 2022-01-20 18:33:55.107342: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.184930 140191300101952 af.py:224] 2022-01-20 18:33:55.107352: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.184971 140191300101952 af.py:224] 2022-01-20 18:33:55.107358: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.185012 140191300101952 af.py:224] 2022-01-20 18:33:55.107366: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.185054 140191300101952 af.py:224] 2022-01-20 18:33:55.107375: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185096 140191300101952 af.py:224] 2022-01-20 18:33:55.107381: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185137 140191300101952 af.py:224] 2022-01-20 18:33:55.107387: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.185178 140191300101952 af.py:224] 2022-01-20 18:33:55.107394: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185219 140191300101952 af.py:224] 2022-01-20 18:33:55.107401: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.185261 140191300101952 af.py:224] 2022-01-20 18:33:55.107413: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.185303 140191300101952 af.py:224] 2022-01-20 18:33:55.107424: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185348 140191300101952 af.py:224] 2022-01-20 18:33:55.107433: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185390 140191300101952 af.py:224] 2022-01-20 18:33:55.107443: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.185431 140191300101952 af.py:224] 2022-01-20 18:33:55.107455: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185473 140191300101952 af.py:224] 2022-01-20 18:33:55.107465: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.185514 140191300101952 af.py:224] 2022-01-20 18:33:55.107480: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.185555 140191300101952 af.py:224] 2022-01-20 18:33:55.107493: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185596 140191300101952 af.py:224] 2022-01-20 18:33:55.107504: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185637 140191300101952 af.py:224] 2022-01-20 18:33:55.107514: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.185693 140191300101952 af.py:224] 2022-01-20 18:33:55.107526: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185736 140191300101952 af.py:224] 2022-01-20 18:33:55.107535: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.185777 140191300101952 af.py:224] 2022-01-20 18:33:55.107550: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.185818 140191300101952 af.py:224] 2022-01-20 18:33:55.107564: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185859 140191300101952 af.py:224] 2022-01-20 18:33:55.107572: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185900 140191300101952 af.py:224] 2022-01-20 18:33:55.107578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.185941 140191300101952 af.py:224] 2022-01-20 18:33:55.107585: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.185982 140191300101952 af.py:224] 2022-01-20 18:33:55.107590: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.186027 140191300101952 af.py:224] 2022-01-20 18:33:55.107599: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.186069 140191300101952 af.py:224] 2022-01-20 18:33:55.107606: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186110 140191300101952 af.py:224] 2022-01-20 18:33:55.107612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186151 140191300101952 af.py:224] 2022-01-20 18:33:55.107620: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.186192 140191300101952 af.py:224] 2022-01-20 18:33:55.107632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186233 140191300101952 af.py:224] 2022-01-20 18:33:55.107642: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.186274 140191300101952 af.py:224] 2022-01-20 18:33:55.107656: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.186315 140191300101952 af.py:224] 2022-01-20 18:33:55.107668: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186356 140191300101952 af.py:224] 2022-01-20 18:33:55.107675: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186398 140191300101952 af.py:224] 2022-01-20 18:33:55.107684: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.186440 140191300101952 af.py:224] 2022-01-20 18:33:55.107696: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186481 140191300101952 af.py:224] 2022-01-20 18:33:55.107705: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.186537 140191300101952 af.py:224] 2022-01-20 18:33:55.107717: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.186578 140191300101952 af.py:224] 2022-01-20 18:33:55.107726: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186619 140191300101952 af.py:224] 2022-01-20 18:33:55.107733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186661 140191300101952 af.py:224] 2022-01-20 18:33:55.107738: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.186701 140191300101952 af.py:224] 2022-01-20 18:33:55.107745: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186744 140191300101952 af.py:224] 2022-01-20 18:33:55.107750: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.186785 140191300101952 af.py:224] 2022-01-20 18:33:55.107759: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.186825 140191300101952 af.py:224] 2022-01-20 18:33:55.107767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186866 140191300101952 af.py:224] 2022-01-20 18:33:55.107777: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186908 140191300101952 af.py:224] 2022-01-20 18:33:55.107787: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.186948 140191300101952 af.py:224] 2022-01-20 18:33:55.107799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.186988 140191300101952 af.py:224] 2022-01-20 18:33:55.107809: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.187029 140191300101952 af.py:224] 2022-01-20 18:33:55.107824: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.187069 140191300101952 af.py:224] 2022-01-20 18:33:55.107915: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187110 140191300101952 af.py:224] 2022-01-20 18:33:55.107926: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187151 140191300101952 af.py:224] 2022-01-20 18:33:55.107934: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.187191 140191300101952 af.py:224] 2022-01-20 18:33:55.107946: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187232 140191300101952 af.py:224] 2022-01-20 18:33:55.107956: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.187272 140191300101952 af.py:224] 2022-01-20 18:33:55.107967: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.187329 140191300101952 af.py:224] 2022-01-20 18:33:55.107975: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187371 140191300101952 af.py:224] 2022-01-20 18:33:55.107982: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187412 140191300101952 af.py:224] 2022-01-20 18:33:55.107987: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.187456 140191300101952 af.py:224] 2022-01-20 18:33:55.107993: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187497 140191300101952 af.py:224] 2022-01-20 18:33:55.107999: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.187537 140191300101952 af.py:224] 2022-01-20 18:33:55.108006: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.187578 140191300101952 af.py:224] 2022-01-20 18:33:55.108013: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187619 140191300101952 af.py:224] 2022-01-20 18:33:55.108020: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187659 140191300101952 af.py:224] 2022-01-20 18:33:55.108025: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.187700 140191300101952 af.py:224] 2022-01-20 18:33:55.108032: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187741 140191300101952 af.py:224] 2022-01-20 18:33:55.108037: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.187782 140191300101952 af.py:224] 2022-01-20 18:33:55.108045: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.187822 140191300101952 af.py:224] 2022-01-20 18:33:55.108052: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187863 140191300101952 af.py:224] 2022-01-20 18:33:55.108059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187903 140191300101952 af.py:224] 2022-01-20 18:33:55.108064: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.187944 140191300101952 af.py:224] 2022-01-20 18:33:55.108071: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.187984 140191300101952 af.py:224] 2022-01-20 18:33:55.108077: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.188024 140191300101952 af.py:224] 2022-01-20 18:33:55.108088: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.188065 140191300101952 af.py:224] 2022-01-20 18:33:55.108096: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188106 140191300101952 af.py:224] 2022-01-20 18:33:55.108103: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188199 140191300101952 af.py:224] 2022-01-20 18:33:55.108109: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.188241 140191300101952 af.py:224] 2022-01-20 18:33:55.108122: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188282 140191300101952 af.py:224] 2022-01-20 18:33:55.108129: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.188323 140191300101952 af.py:224] 2022-01-20 18:33:55.108139: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.188364 140191300101952 af.py:224] 2022-01-20 18:33:55.108149: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188405 140191300101952 af.py:224] 2022-01-20 18:33:55.108156: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188446 140191300101952 af.py:224] 2022-01-20 18:33:55.108161: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.188487 140191300101952 af.py:224] 2022-01-20 18:33:55.108168: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188527 140191300101952 af.py:224] 2022-01-20 18:33:55.108173: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.188568 140191300101952 af.py:224] 2022-01-20 18:33:55.108181: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.188608 140191300101952 af.py:224] 2022-01-20 18:33:55.108188: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188649 140191300101952 af.py:224] 2022-01-20 18:33:55.108195: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188690 140191300101952 af.py:224] 2022-01-20 18:33:55.108203: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.188731 140191300101952 af.py:224] 2022-01-20 18:33:55.108214: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188772 140191300101952 af.py:224] 2022-01-20 18:33:55.108223: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.188813 140191300101952 af.py:224] 2022-01-20 18:33:55.108238: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.188857 140191300101952 af.py:224] 2022-01-20 18:33:55.108249: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188899 140191300101952 af.py:224] 2022-01-20 18:33:55.108256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.188940 140191300101952 af.py:224] 2022-01-20 18:33:55.108262: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.189000 140191300101952 af.py:224] 2022-01-20 18:33:55.108273: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189060 140191300101952 af.py:224] 2022-01-20 18:33:55.108282: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.189102 140191300101952 af.py:224] 2022-01-20 18:33:55.108297: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.189144 140191300101952 af.py:224] 2022-01-20 18:33:55.108309: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189186 140191300101952 af.py:224] 2022-01-20 18:33:55.108320: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189228 140191300101952 af.py:224] 2022-01-20 18:33:55.108328: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.189269 140191300101952 af.py:224] 2022-01-20 18:33:55.108340: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189326 140191300101952 af.py:224] 2022-01-20 18:33:55.108348: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.189367 140191300101952 af.py:224] 2022-01-20 18:33:55.108363: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.189408 140191300101952 af.py:224] 2022-01-20 18:33:55.108377: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189448 140191300101952 af.py:224] 2022-01-20 18:33:55.108388: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189489 140191300101952 af.py:224] 2022-01-20 18:33:55.108397: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.189530 140191300101952 af.py:224] 2022-01-20 18:33:55.108405: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189571 140191300101952 af.py:224] 2022-01-20 18:33:55.108410: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.189615 140191300101952 af.py:224] 2022-01-20 18:33:55.108419: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.189656 140191300101952 af.py:224] 2022-01-20 18:33:55.108431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189698 140191300101952 af.py:224] 2022-01-20 18:33:55.108442: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189738 140191300101952 af.py:224] 2022-01-20 18:33:55.108450: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.189779 140191300101952 af.py:224] 2022-01-20 18:33:55.108462: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189819 140191300101952 af.py:224] 2022-01-20 18:33:55.108471: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.189859 140191300101952 af.py:224] 2022-01-20 18:33:55.108485: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.189899 140191300101952 af.py:224] 2022-01-20 18:33:55.108498: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189939 140191300101952 af.py:224] 2022-01-20 18:33:55.108509: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.189979 140191300101952 af.py:224] 2022-01-20 18:33:55.108517: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.190019 140191300101952 af.py:224] 2022-01-20 18:33:55.108524: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190061 140191300101952 af.py:224] 2022-01-20 18:33:55.108529: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.190102 140191300101952 af.py:224] 2022-01-20 18:33:55.108538: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.190142 140191300101952 af.py:224] 2022-01-20 18:33:55.108546: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190183 140191300101952 af.py:224] 2022-01-20 18:33:55.108554: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190224 140191300101952 af.py:224] 2022-01-20 18:33:55.108562: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.190264 140191300101952 af.py:224] 2022-01-20 18:33:55.108574: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190308 140191300101952 af.py:224] 2022-01-20 18:33:55.108582: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.190349 140191300101952 af.py:224] 2022-01-20 18:33:55.108596: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.190389 140191300101952 af.py:224] 2022-01-20 18:33:55.108608: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190429 140191300101952 af.py:224] 2022-01-20 18:33:55.108619: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190470 140191300101952 af.py:224] 2022-01-20 18:33:55.108625: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.190511 140191300101952 af.py:224] 2022-01-20 18:33:55.108632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190551 140191300101952 af.py:224] 2022-01-20 18:33:55.108637: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.190592 140191300101952 af.py:224] 2022-01-20 18:33:55.108646: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.190633 140191300101952 af.py:224] 2022-01-20 18:33:55.108653: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190690 140191300101952 af.py:224] 2022-01-20 18:33:55.108660: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190732 140191300101952 af.py:224] 2022-01-20 18:33:55.108664: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.190773 140191300101952 af.py:224] 2022-01-20 18:33:55.108671: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190814 140191300101952 af.py:224] 2022-01-20 18:33:55.108677: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.190855 140191300101952 af.py:224] 2022-01-20 18:33:55.108685: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.190895 140191300101952 af.py:224] 2022-01-20 18:33:55.108691: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190935 140191300101952 af.py:224] 2022-01-20 18:33:55.108698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.190976 140191300101952 af.py:224] 2022-01-20 18:33:55.108703: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.191020 140191300101952 af.py:224] 2022-01-20 18:33:55.108709: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191061 140191300101952 af.py:224] 2022-01-20 18:33:55.108714: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.191102 140191300101952 af.py:224] 2022-01-20 18:33:55.108722: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.191143 140191300101952 af.py:224] 2022-01-20 18:33:55.108729: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191185 140191300101952 af.py:224] 2022-01-20 18:33:55.108736: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191226 140191300101952 af.py:224] 2022-01-20 18:33:55.108741: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.191267 140191300101952 af.py:224] 2022-01-20 18:33:55.108747: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191308 140191300101952 af.py:224] 2022-01-20 18:33:55.108752: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.191348 140191300101952 af.py:224] 2022-01-20 18:33:55.108760: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.191390 140191300101952 af.py:224] 2022-01-20 18:33:55.108767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191431 140191300101952 af.py:224] 2022-01-20 18:33:55.108773: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191472 140191300101952 af.py:224] 2022-01-20 18:33:55.108778: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.191513 140191300101952 af.py:224] 2022-01-20 18:33:55.108785: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191554 140191300101952 af.py:224] 2022-01-20 18:33:55.108790: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.191595 140191300101952 af.py:224] 2022-01-20 18:33:55.108798: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.191636 140191300101952 af.py:224] 2022-01-20 18:33:55.108804: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191676 140191300101952 af.py:224] 2022-01-20 18:33:55.108811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191720 140191300101952 af.py:224] 2022-01-20 18:33:55.108816: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.191761 140191300101952 af.py:224] 2022-01-20 18:33:55.108840: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191802 140191300101952 af.py:224] 2022-01-20 18:33:55.108850: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.191843 140191300101952 af.py:224] 2022-01-20 18:33:55.108864: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.191884 140191300101952 af.py:224] 2022-01-20 18:33:55.108877: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191924 140191300101952 af.py:224] 2022-01-20 18:33:55.108888: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.191966 140191300101952 af.py:224] 2022-01-20 18:33:55.108896: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.192006 140191300101952 af.py:224] 2022-01-20 18:33:55.108908: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192047 140191300101952 af.py:224] 2022-01-20 18:33:55.108917: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.192088 140191300101952 af.py:224] 2022-01-20 18:33:55.108931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.192134 140191300101952 af.py:224] 2022-01-20 18:33:55.108943: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192226 140191300101952 af.py:224] 2022-01-20 18:33:55.108954: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192268 140191300101952 af.py:224] 2022-01-20 18:33:55.108962: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.192308 140191300101952 af.py:224] 2022-01-20 18:33:55.108974: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192363 140191300101952 af.py:224] 2022-01-20 18:33:55.108982: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.192405 140191300101952 af.py:224] 2022-01-20 18:33:55.108997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.192450 140191300101952 af.py:224] 2022-01-20 18:33:55.109008: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192490 140191300101952 af.py:224] 2022-01-20 18:33:55.109018: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192530 140191300101952 af.py:224] 2022-01-20 18:33:55.109027: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.192571 140191300101952 af.py:224] 2022-01-20 18:33:55.109037: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192611 140191300101952 af.py:224] 2022-01-20 18:33:55.109046: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.192651 140191300101952 af.py:224] 2022-01-20 18:33:55.109060: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.192692 140191300101952 af.py:224] 2022-01-20 18:33:55.109071: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192732 140191300101952 af.py:224] 2022-01-20 18:33:55.109081: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192773 140191300101952 af.py:224] 2022-01-20 18:33:55.109090: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.192813 140191300101952 af.py:224] 2022-01-20 18:33:55.109101: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192854 140191300101952 af.py:224] 2022-01-20 18:33:55.109109: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.192894 140191300101952 af.py:224] 2022-01-20 18:33:55.109124: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.192935 140191300101952 af.py:224] 2022-01-20 18:33:55.109136: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.192975 140191300101952 af.py:224] 2022-01-20 18:33:55.109146: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193016 140191300101952 af.py:224] 2022-01-20 18:33:55.109154: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.193056 140191300101952 af.py:224] 2022-01-20 18:33:55.109166: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193096 140191300101952 af.py:224] 2022-01-20 18:33:55.109176: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.193140 140191300101952 af.py:224] 2022-01-20 18:33:55.109191: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.193181 140191300101952 af.py:224] 2022-01-20 18:33:55.109206: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193221 140191300101952 af.py:224] 2022-01-20 18:33:55.109218: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193262 140191300101952 af.py:224] 2022-01-20 18:33:55.109228: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.193302 140191300101952 af.py:224] 2022-01-20 18:33:55.109240: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193343 140191300101952 af.py:224] 2022-01-20 18:33:55.109249: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.193384 140191300101952 af.py:224] 2022-01-20 18:33:55.109264: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.193424 140191300101952 af.py:224] 2022-01-20 18:33:55.109278: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193465 140191300101952 af.py:224] 2022-01-20 18:33:55.109289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193506 140191300101952 af.py:224] 2022-01-20 18:33:55.109298: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.193547 140191300101952 af.py:224] 2022-01-20 18:33:55.109310: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193588 140191300101952 af.py:224] 2022-01-20 18:33:55.109319: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.193628 140191300101952 af.py:224] 2022-01-20 18:33:55.109332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.193669 140191300101952 af.py:224] 2022-01-20 18:33:55.109345: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193710 140191300101952 af.py:224] 2022-01-20 18:33:55.109356: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193751 140191300101952 af.py:224] 2022-01-20 18:33:55.109366: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.193792 140191300101952 af.py:224] 2022-01-20 18:33:55.109378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193835 140191300101952 af.py:224] 2022-01-20 18:33:55.109387: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.193876 140191300101952 af.py:224] 2022-01-20 18:33:55.109402: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.193917 140191300101952 af.py:224] 2022-01-20 18:33:55.109412: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.193972 140191300101952 af.py:224] 2022-01-20 18:33:55.109419: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194014 140191300101952 af.py:224] 2022-01-20 18:33:55.109425: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.194055 140191300101952 af.py:224] 2022-01-20 18:33:55.109431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194095 140191300101952 af.py:224] 2022-01-20 18:33:55.109437: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.194136 140191300101952 af.py:224] 2022-01-20 18:33:55.109445: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.194177 140191300101952 af.py:224] 2022-01-20 18:33:55.109452: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194218 140191300101952 af.py:224] 2022-01-20 18:33:55.109461: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194259 140191300101952 af.py:224] 2022-01-20 18:33:55.109467: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.194300 140191300101952 af.py:224] 2022-01-20 18:33:55.109474: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194341 140191300101952 af.py:224] 2022-01-20 18:33:55.109479: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.194381 140191300101952 af.py:224] 2022-01-20 18:33:55.109488: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.194422 140191300101952 af.py:224] 2022-01-20 18:33:55.109496: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194463 140191300101952 af.py:224] 2022-01-20 18:33:55.109502: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194508 140191300101952 af.py:224] 2022-01-20 18:33:55.109507: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.194549 140191300101952 af.py:224] 2022-01-20 18:33:55.109516: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194589 140191300101952 af.py:224] 2022-01-20 18:33:55.109525: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.194630 140191300101952 af.py:224] 2022-01-20 18:33:55.109540: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.194671 140191300101952 af.py:224] 2022-01-20 18:33:55.109553: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194711 140191300101952 af.py:224] 2022-01-20 18:33:55.109564: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194752 140191300101952 af.py:224] 2022-01-20 18:33:55.109573: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.194792 140191300101952 af.py:224] 2022-01-20 18:33:55.109584: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194833 140191300101952 af.py:224] 2022-01-20 18:33:55.109589: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.194874 140191300101952 af.py:224] 2022-01-20 18:33:55.109598: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.194915 140191300101952 af.py:224] 2022-01-20 18:33:55.109605: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194956 140191300101952 af.py:224] 2022-01-20 18:33:55.109612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.194996 140191300101952 af.py:224] 2022-01-20 18:33:55.109620: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.195037 140191300101952 af.py:224] 2022-01-20 18:33:55.109630: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195078 140191300101952 af.py:224] 2022-01-20 18:33:55.109640: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.195119 140191300101952 af.py:224] 2022-01-20 18:33:55.109655: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.195159 140191300101952 af.py:224] 2022-01-20 18:33:55.109668: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195202 140191300101952 af.py:224] 2022-01-20 18:33:55.109680: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195244 140191300101952 af.py:224] 2022-01-20 18:33:55.109690: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.195285 140191300101952 af.py:224] 2022-01-20 18:33:55.109702: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195325 140191300101952 af.py:224] 2022-01-20 18:33:55.109711: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.195365 140191300101952 af.py:224] 2022-01-20 18:33:55.109721: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.195406 140191300101952 af.py:224] 2022-01-20 18:33:55.109728: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195446 140191300101952 af.py:224] 2022-01-20 18:33:55.109735: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195487 140191300101952 af.py:224] 2022-01-20 18:33:55.109740: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.195527 140191300101952 af.py:224] 2022-01-20 18:33:55.109747: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195584 140191300101952 af.py:224] 2022-01-20 18:33:55.109752: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.195626 140191300101952 af.py:224] 2022-01-20 18:33:55.109760: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.195693 140191300101952 af.py:224] 2022-01-20 18:33:55.109767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195734 140191300101952 af.py:224] 2022-01-20 18:33:55.109773: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195776 140191300101952 af.py:224] 2022-01-20 18:33:55.109778: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.195817 140191300101952 af.py:224] 2022-01-20 18:33:55.109785: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195859 140191300101952 af.py:224] 2022-01-20 18:33:55.109790: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.195900 140191300101952 af.py:224] 2022-01-20 18:33:55.109799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.195945 140191300101952 af.py:224] 2022-01-20 18:33:55.109806: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.195987 140191300101952 af.py:224] 2022-01-20 18:33:55.109812: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196028 140191300101952 af.py:224] 2022-01-20 18:33:55.109817: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.196069 140191300101952 af.py:224] 2022-01-20 18:33:55.109823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196110 140191300101952 af.py:224] 2022-01-20 18:33:55.109829: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.196157 140191300101952 af.py:224] 2022-01-20 18:33:55.109837: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.196213 140191300101952 af.py:224] 2022-01-20 18:33:55.109844: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196254 140191300101952 af.py:224] 2022-01-20 18:33:55.109850: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196295 140191300101952 af.py:224] 2022-01-20 18:33:55.109855: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.196335 140191300101952 af.py:224] 2022-01-20 18:33:55.109862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196376 140191300101952 af.py:224] 2022-01-20 18:33:55.109867: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.196417 140191300101952 af.py:224] 2022-01-20 18:33:55.109875: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.196458 140191300101952 af.py:224] 2022-01-20 18:33:55.109882: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196499 140191300101952 af.py:224] 2022-01-20 18:33:55.109888: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196539 140191300101952 af.py:224] 2022-01-20 18:33:55.109893: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.196580 140191300101952 af.py:224] 2022-01-20 18:33:55.109900: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196620 140191300101952 af.py:224] 2022-01-20 18:33:55.109906: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.196666 140191300101952 af.py:224] 2022-01-20 18:33:55.109914: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.196708 140191300101952 af.py:224] 2022-01-20 18:33:55.109921: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196748 140191300101952 af.py:224] 2022-01-20 18:33:55.109928: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196789 140191300101952 af.py:224] 2022-01-20 18:33:55.109933: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.196829 140191300101952 af.py:224] 2022-01-20 18:33:55.109939: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196870 140191300101952 af.py:224] 2022-01-20 18:33:55.109944: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.196912 140191300101952 af.py:224] 2022-01-20 18:33:55.109953: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.196952 140191300101952 af.py:224] 2022-01-20 18:33:55.109961: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.196993 140191300101952 af.py:224] 2022-01-20 18:33:55.109967: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197033 140191300101952 af.py:224] 2022-01-20 18:33:55.109972: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.197073 140191300101952 af.py:224] 2022-01-20 18:33:55.109979: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197114 140191300101952 af.py:224] 2022-01-20 18:33:55.109984: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.197154 140191300101952 af.py:224] 2022-01-20 18:33:55.109992: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.197195 140191300101952 af.py:224] 2022-01-20 18:33:55.109999: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197250 140191300101952 af.py:224] 2022-01-20 18:33:55.110005: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197291 140191300101952 af.py:224] 2022-01-20 18:33:55.110010: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.197332 140191300101952 af.py:224] 2022-01-20 18:33:55.110016: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197376 140191300101952 af.py:224] 2022-01-20 18:33:55.110021: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.197417 140191300101952 af.py:224] 2022-01-20 18:33:55.110029: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.197457 140191300101952 af.py:224] 2022-01-20 18:33:55.110036: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197498 140191300101952 af.py:224] 2022-01-20 18:33:55.110042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197539 140191300101952 af.py:224] 2022-01-20 18:33:55.110047: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.197579 140191300101952 af.py:224] 2022-01-20 18:33:55.110054: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197619 140191300101952 af.py:224] 2022-01-20 18:33:55.110059: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.197660 140191300101952 af.py:224] 2022-01-20 18:33:55.110067: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.197700 140191300101952 af.py:224] 2022-01-20 18:33:55.110073: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197740 140191300101952 af.py:224] 2022-01-20 18:33:55.110080: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197781 140191300101952 af.py:224] 2022-01-20 18:33:55.110085: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.197821 140191300101952 af.py:224] 2022-01-20 18:33:55.110092: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197862 140191300101952 af.py:224] 2022-01-20 18:33:55.110097: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.197903 140191300101952 af.py:224] 2022-01-20 18:33:55.110105: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.197943 140191300101952 af.py:224] 2022-01-20 18:33:55.110112: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.197983 140191300101952 af.py:224] 2022-01-20 18:33:55.110118: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198027 140191300101952 af.py:224] 2022-01-20 18:33:55.110123: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.198068 140191300101952 af.py:224] 2022-01-20 18:33:55.110130: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198108 140191300101952 af.py:224] 2022-01-20 18:33:55.110135: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.198149 140191300101952 af.py:224] 2022-01-20 18:33:55.110144: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.198189 140191300101952 af.py:224] 2022-01-20 18:33:55.110151: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198230 140191300101952 af.py:224] 2022-01-20 18:33:55.110157: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198271 140191300101952 af.py:224] 2022-01-20 18:33:55.110162: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.198312 140191300101952 af.py:224] 2022-01-20 18:33:55.110168: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198353 140191300101952 af.py:224] 2022-01-20 18:33:55.110173: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.198394 140191300101952 af.py:224] 2022-01-20 18:33:55.110181: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.198434 140191300101952 af.py:224] 2022-01-20 18:33:55.110188: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198475 140191300101952 af.py:224] 2022-01-20 18:33:55.110194: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198516 140191300101952 af.py:224] 2022-01-20 18:33:55.110199: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.198556 140191300101952 af.py:224] 2022-01-20 18:33:55.110206: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198597 140191300101952 af.py:224] 2022-01-20 18:33:55.110211: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.198637 140191300101952 af.py:224] 2022-01-20 18:33:55.110219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.198678 140191300101952 af.py:224] 2022-01-20 18:33:55.110226: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198722 140191300101952 af.py:224] 2022-01-20 18:33:55.110232: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198762 140191300101952 af.py:224] 2022-01-20 18:33:55.110237: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.198804 140191300101952 af.py:224] 2022-01-20 18:33:55.110243: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198845 140191300101952 af.py:224] 2022-01-20 18:33:55.110247: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.198899 140191300101952 af.py:224] 2022-01-20 18:33:55.110256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.198940 140191300101952 af.py:224] 2022-01-20 18:33:55.110263: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.198982 140191300101952 af.py:224] 2022-01-20 18:33:55.110271: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199023 140191300101952 af.py:224] 2022-01-20 18:33:55.110280: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.199064 140191300101952 af.py:224] 2022-01-20 18:33:55.110292: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199104 140191300101952 af.py:224] 2022-01-20 18:33:55.110302: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.199145 140191300101952 af.py:224] 2022-01-20 18:33:55.110317: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.199186 140191300101952 af.py:224] 2022-01-20 18:33:55.110327: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199226 140191300101952 af.py:224] 2022-01-20 18:33:55.110335: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199267 140191300101952 af.py:224] 2022-01-20 18:33:55.110341: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.199308 140191300101952 af.py:224] 2022-01-20 18:33:55.110347: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199349 140191300101952 af.py:224] 2022-01-20 18:33:55.110352: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.199390 140191300101952 af.py:224] 2022-01-20 18:33:55.110361: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.199434 140191300101952 af.py:224] 2022-01-20 18:33:55.110370: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199475 140191300101952 af.py:224] 2022-01-20 18:33:55.110378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199515 140191300101952 af.py:224] 2022-01-20 18:33:55.110387: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.199556 140191300101952 af.py:224] 2022-01-20 18:33:55.110399: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199597 140191300101952 af.py:224] 2022-01-20 18:33:55.110408: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.199638 140191300101952 af.py:224] 2022-01-20 18:33:55.110424: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.199679 140191300101952 af.py:224] 2022-01-20 18:33:55.110436: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199720 140191300101952 af.py:224] 2022-01-20 18:33:55.110445: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199761 140191300101952 af.py:224] 2022-01-20 18:33:55.110450: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.199802 140191300101952 af.py:224] 2022-01-20 18:33:55.110457: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199843 140191300101952 af.py:224] 2022-01-20 18:33:55.110462: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.199883 140191300101952 af.py:224] 2022-01-20 18:33:55.110470: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.199924 140191300101952 af.py:224] 2022-01-20 18:33:55.110478: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.199965 140191300101952 af.py:224] 2022-01-20 18:33:55.110484: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200005 140191300101952 af.py:224] 2022-01-20 18:33:55.110492: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.200047 140191300101952 af.py:224] 2022-01-20 18:33:55.110503: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200088 140191300101952 af.py:224] 2022-01-20 18:33:55.110512: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.200136 140191300101952 af.py:224] 2022-01-20 18:33:55.110527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.200230 140191300101952 af.py:224] 2022-01-20 18:33:55.110540: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200270 140191300101952 af.py:224] 2022-01-20 18:33:55.110551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200310 140191300101952 af.py:224] 2022-01-20 18:33:55.110560: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.200351 140191300101952 af.py:224] 2022-01-20 18:33:55.110569: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200392 140191300101952 af.py:224] 2022-01-20 18:33:55.110578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.200432 140191300101952 af.py:224] 2022-01-20 18:33:55.110593: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.200472 140191300101952 af.py:224] 2022-01-20 18:33:55.110605: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200513 140191300101952 af.py:224] 2022-01-20 18:33:55.110613: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200568 140191300101952 af.py:224] 2022-01-20 18:33:55.110618: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.200610 140191300101952 af.py:224] 2022-01-20 18:33:55.110625: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200651 140191300101952 af.py:224] 2022-01-20 18:33:55.110630: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.200691 140191300101952 af.py:224] 2022-01-20 18:33:55.110639: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.200731 140191300101952 af.py:224] 2022-01-20 18:33:55.110647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200772 140191300101952 af.py:224] 2022-01-20 18:33:55.110654: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200812 140191300101952 af.py:224] 2022-01-20 18:33:55.110663: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.200853 140191300101952 af.py:224] 2022-01-20 18:33:55.110676: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.200897 140191300101952 af.py:224] 2022-01-20 18:33:55.110686: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.200938 140191300101952 af.py:224] 2022-01-20 18:33:55.110701: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.200979 140191300101952 af.py:224] 2022-01-20 18:33:55.110712: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201020 140191300101952 af.py:224] 2022-01-20 18:33:55.110722: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201060 140191300101952 af.py:224] 2022-01-20 18:33:55.110732: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.201101 140191300101952 af.py:224] 2022-01-20 18:33:55.110743: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201142 140191300101952 af.py:224] 2022-01-20 18:33:55.110752: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.201183 140191300101952 af.py:224] 2022-01-20 18:33:55.110805: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.201224 140191300101952 af.py:224] 2022-01-20 18:33:55.110818: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201264 140191300101952 af.py:224] 2022-01-20 18:33:55.110828: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201305 140191300101952 af.py:224] 2022-01-20 18:33:55.110837: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.201346 140191300101952 af.py:224] 2022-01-20 18:33:55.110849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201387 140191300101952 af.py:224] 2022-01-20 18:33:55.110857: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.201428 140191300101952 af.py:224] 2022-01-20 18:33:55.110870: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.201468 140191300101952 af.py:224] 2022-01-20 18:33:55.110883: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201508 140191300101952 af.py:224] 2022-01-20 18:33:55.110893: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201552 140191300101952 af.py:224] 2022-01-20 18:33:55.110902: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.201593 140191300101952 af.py:224] 2022-01-20 18:33:55.110914: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201633 140191300101952 af.py:224] 2022-01-20 18:33:55.110924: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.201674 140191300101952 af.py:224] 2022-01-20 18:33:55.110954: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.201714 140191300101952 af.py:224] 2022-01-20 18:33:55.110967: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201755 140191300101952 af.py:224] 2022-01-20 18:33:55.110976: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201796 140191300101952 af.py:224] 2022-01-20 18:33:55.110984: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.201836 140191300101952 af.py:224] 2022-01-20 18:33:55.110996: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201877 140191300101952 af.py:224] 2022-01-20 18:33:55.111004: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.201918 140191300101952 af.py:224] 2022-01-20 18:33:55.111018: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.201958 140191300101952 af.py:224] 2022-01-20 18:33:55.111031: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.201999 140191300101952 af.py:224] 2022-01-20 18:33:55.111040: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202039 140191300101952 af.py:224] 2022-01-20 18:33:55.111046: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.202080 140191300101952 af.py:224] 2022-01-20 18:33:55.111053: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202121 140191300101952 af.py:224] 2022-01-20 18:33:55.111058: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.202180 140191300101952 af.py:224] 2022-01-20 18:33:55.111065: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.202223 140191300101952 af.py:224] 2022-01-20 18:33:55.111072: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202284 140191300101952 af.py:224] 2022-01-20 18:33:55.111079: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202326 140191300101952 af.py:224] 2022-01-20 18:33:55.111085: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.202368 140191300101952 af.py:224] 2022-01-20 18:33:55.111094: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202409 140191300101952 af.py:224] 2022-01-20 18:33:55.111099: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.202465 140191300101952 af.py:224] 2022-01-20 18:33:55.111107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.202505 140191300101952 af.py:224] 2022-01-20 18:33:55.111115: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202546 140191300101952 af.py:224] 2022-01-20 18:33:55.111122: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202587 140191300101952 af.py:224] 2022-01-20 18:33:55.111127: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.202628 140191300101952 af.py:224] 2022-01-20 18:33:55.111134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202668 140191300101952 af.py:224] 2022-01-20 18:33:55.111140: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.202708 140191300101952 af.py:224] 2022-01-20 18:33:55.111152: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.202749 140191300101952 af.py:224] 2022-01-20 18:33:55.111162: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202789 140191300101952 af.py:224] 2022-01-20 18:33:55.111173: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202830 140191300101952 af.py:224] 2022-01-20 18:33:55.111182: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.202871 140191300101952 af.py:224] 2022-01-20 18:33:55.111195: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.202911 140191300101952 af.py:224] 2022-01-20 18:33:55.111204: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.202952 140191300101952 af.py:224] 2022-01-20 18:33:55.111219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.202996 140191300101952 af.py:224] 2022-01-20 18:33:55.111231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203037 140191300101952 af.py:224] 2022-01-20 18:33:55.111243: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203078 140191300101952 af.py:224] 2022-01-20 18:33:55.111252: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.203118 140191300101952 af.py:224] 2022-01-20 18:33:55.111264: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203159 140191300101952 af.py:224] 2022-01-20 18:33:55.111273: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.203199 140191300101952 af.py:224] 2022-01-20 18:33:55.111289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.203240 140191300101952 af.py:224] 2022-01-20 18:33:55.111301: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203281 140191300101952 af.py:224] 2022-01-20 18:33:55.111311: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203321 140191300101952 af.py:224] 2022-01-20 18:33:55.111320: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.203362 140191300101952 af.py:224] 2022-01-20 18:33:55.111332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203402 140191300101952 af.py:224] 2022-01-20 18:33:55.111342: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.203443 140191300101952 af.py:224] 2022-01-20 18:33:55.111354: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.203484 140191300101952 af.py:224] 2022-01-20 18:33:55.111364: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203524 140191300101952 af.py:224] 2022-01-20 18:33:55.111371: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203564 140191300101952 af.py:224] 2022-01-20 18:33:55.111376: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.203604 140191300101952 af.py:224] 2022-01-20 18:33:55.111383: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203645 140191300101952 af.py:224] 2022-01-20 18:33:55.111388: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.203689 140191300101952 af.py:224] 2022-01-20 18:33:55.111399: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.203730 140191300101952 af.py:224] 2022-01-20 18:33:55.111410: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203770 140191300101952 af.py:224] 2022-01-20 18:33:55.111416: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203828 140191300101952 af.py:224] 2022-01-20 18:33:55.111422: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.203870 140191300101952 af.py:224] 2022-01-20 18:33:55.111431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.203911 140191300101952 af.py:224] 2022-01-20 18:33:55.111438: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.203951 140191300101952 af.py:224] 2022-01-20 18:33:55.111448: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.203991 140191300101952 af.py:224] 2022-01-20 18:33:55.111456: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204032 140191300101952 af.py:224] 2022-01-20 18:33:55.111462: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204073 140191300101952 af.py:224] 2022-01-20 18:33:55.111468: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.204117 140191300101952 af.py:224] 2022-01-20 18:33:55.111474: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204194 140191300101952 af.py:224] 2022-01-20 18:33:55.111479: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.204249 140191300101952 af.py:224] 2022-01-20 18:33:55.111487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.204290 140191300101952 af.py:224] 2022-01-20 18:33:55.111495: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204331 140191300101952 af.py:224] 2022-01-20 18:33:55.111503: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204372 140191300101952 af.py:224] 2022-01-20 18:33:55.111512: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.204413 140191300101952 af.py:224] 2022-01-20 18:33:55.111524: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204458 140191300101952 af.py:224] 2022-01-20 18:33:55.111532: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.204500 140191300101952 af.py:224] 2022-01-20 18:33:55.111541: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.204541 140191300101952 af.py:224] 2022-01-20 18:33:55.111550: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204582 140191300101952 af.py:224] 2022-01-20 18:33:55.111557: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204623 140191300101952 af.py:224] 2022-01-20 18:33:55.111562: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.204663 140191300101952 af.py:224] 2022-01-20 18:33:55.111569: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204704 140191300101952 af.py:224] 2022-01-20 18:33:55.111574: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.204744 140191300101952 af.py:224] 2022-01-20 18:33:55.111585: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.204785 140191300101952 af.py:224] 2022-01-20 18:33:55.111597: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204825 140191300101952 af.py:224] 2022-01-20 18:33:55.111608: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204866 140191300101952 af.py:224] 2022-01-20 18:33:55.111618: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.204907 140191300101952 af.py:224] 2022-01-20 18:33:55.111630: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.204948 140191300101952 af.py:224] 2022-01-20 18:33:55.111639: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.204988 140191300101952 af.py:224] 2022-01-20 18:33:55.111654: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.205029 140191300101952 af.py:224] 2022-01-20 18:33:55.111667: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205069 140191300101952 af.py:224] 2022-01-20 18:33:55.111677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205113 140191300101952 af.py:224] 2022-01-20 18:33:55.111685: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.205154 140191300101952 af.py:224] 2022-01-20 18:33:55.111698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205194 140191300101952 af.py:224] 2022-01-20 18:33:55.111706: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.205235 140191300101952 af.py:224] 2022-01-20 18:33:55.111720: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.205276 140191300101952 af.py:224] 2022-01-20 18:33:55.111733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205317 140191300101952 af.py:224] 2022-01-20 18:33:55.111741: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205357 140191300101952 af.py:224] 2022-01-20 18:33:55.111746: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.205398 140191300101952 af.py:224] 2022-01-20 18:33:55.111753: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205438 140191300101952 af.py:224] 2022-01-20 18:33:55.111758: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.205479 140191300101952 af.py:224] 2022-01-20 18:33:55.111767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.205535 140191300101952 af.py:224] 2022-01-20 18:33:55.111774: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205577 140191300101952 af.py:224] 2022-01-20 18:33:55.111780: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205618 140191300101952 af.py:224] 2022-01-20 18:33:55.111785: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.205659 140191300101952 af.py:224] 2022-01-20 18:33:55.111795: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205700 140191300101952 af.py:224] 2022-01-20 18:33:55.111804: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.205741 140191300101952 af.py:224] 2022-01-20 18:33:55.111816: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.205781 140191300101952 af.py:224] 2022-01-20 18:33:55.111823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205825 140191300101952 af.py:224] 2022-01-20 18:33:55.111830: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205866 140191300101952 af.py:224] 2022-01-20 18:33:55.111835: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.205906 140191300101952 af.py:224] 2022-01-20 18:33:55.111842: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.205946 140191300101952 af.py:224] 2022-01-20 18:33:55.111849: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.205986 140191300101952 af.py:224] 2022-01-20 18:33:55.111859: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.206027 140191300101952 af.py:224] 2022-01-20 18:33:55.111867: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206068 140191300101952 af.py:224] 2022-01-20 18:33:55.111873: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206109 140191300101952 af.py:224] 2022-01-20 18:33:55.111878: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.206149 140191300101952 af.py:224] 2022-01-20 18:33:55.111885: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206189 140191300101952 af.py:224] 2022-01-20 18:33:55.111890: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.206230 140191300101952 af.py:224] 2022-01-20 18:33:55.111899: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.206270 140191300101952 af.py:224] 2022-01-20 18:33:55.111908: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206310 140191300101952 af.py:224] 2022-01-20 18:33:55.111917: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206351 140191300101952 af.py:224] 2022-01-20 18:33:55.111923: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.206391 140191300101952 af.py:224] 2022-01-20 18:33:55.111930: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206433 140191300101952 af.py:224] 2022-01-20 18:33:55.111935: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.206473 140191300101952 af.py:224] 2022-01-20 18:33:55.111943: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.206516 140191300101952 af.py:224] 2022-01-20 18:33:55.111950: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206558 140191300101952 af.py:224] 2022-01-20 18:33:55.111957: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206599 140191300101952 af.py:224] 2022-01-20 18:33:55.111962: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.206639 140191300101952 af.py:224] 2022-01-20 18:33:55.111971: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206680 140191300101952 af.py:224] 2022-01-20 18:33:55.111978: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.206721 140191300101952 af.py:224] 2022-01-20 18:33:55.111990: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.206762 140191300101952 af.py:224] 2022-01-20 18:33:55.112002: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206802 140191300101952 af.py:224] 2022-01-20 18:33:55.112013: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206843 140191300101952 af.py:224] 2022-01-20 18:33:55.112023: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.206884 140191300101952 af.py:224] 2022-01-20 18:33:55.112033: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.206924 140191300101952 af.py:224] 2022-01-20 18:33:55.112038: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.206965 140191300101952 af.py:224] 2022-01-20 18:33:55.112047: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.207005 140191300101952 af.py:224] 2022-01-20 18:33:55.112055: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207046 140191300101952 af.py:224] 2022-01-20 18:33:55.112062: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207086 140191300101952 af.py:224] 2022-01-20 18:33:55.112067: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.207141 140191300101952 af.py:224] 2022-01-20 18:33:55.112074: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207183 140191300101952 af.py:224] 2022-01-20 18:33:55.112082: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.207228 140191300101952 af.py:224] 2022-01-20 18:33:55.112095: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.207270 140191300101952 af.py:224] 2022-01-20 18:33:55.112107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207310 140191300101952 af.py:224] 2022-01-20 18:33:55.112121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207351 140191300101952 af.py:224] 2022-01-20 18:33:55.112129: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.207392 140191300101952 af.py:224] 2022-01-20 18:33:55.112139: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207433 140191300101952 af.py:224] 2022-01-20 18:33:55.112149: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.207474 140191300101952 af.py:224] 2022-01-20 18:33:55.112164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.207515 140191300101952 af.py:224] 2022-01-20 18:33:55.112179: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207556 140191300101952 af.py:224] 2022-01-20 18:33:55.112191: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207597 140191300101952 af.py:224] 2022-01-20 18:33:55.112198: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.207638 140191300101952 af.py:224] 2022-01-20 18:33:55.112205: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207678 140191300101952 af.py:224] 2022-01-20 18:33:55.112210: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.207719 140191300101952 af.py:224] 2022-01-20 18:33:55.112218: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.207759 140191300101952 af.py:224] 2022-01-20 18:33:55.112226: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207800 140191300101952 af.py:224] 2022-01-20 18:33:55.112233: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207841 140191300101952 af.py:224] 2022-01-20 18:33:55.112238: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.207881 140191300101952 af.py:224] 2022-01-20 18:33:55.112247: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.207926 140191300101952 af.py:224] 2022-01-20 18:33:55.112256: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.207967 140191300101952 af.py:224] 2022-01-20 18:33:55.112269: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.208008 140191300101952 af.py:224] 2022-01-20 18:33:55.112278: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208048 140191300101952 af.py:224] 2022-01-20 18:33:55.112284: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208089 140191300101952 af.py:224] 2022-01-20 18:33:55.112290: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.208154 140191300101952 af.py:224] 2022-01-20 18:33:55.112297: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208217 140191300101952 af.py:224] 2022-01-20 18:33:55.112302: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.208258 140191300101952 af.py:224] 2022-01-20 18:33:55.112310: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.208299 140191300101952 af.py:224] 2022-01-20 18:33:55.112321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208339 140191300101952 af.py:224] 2022-01-20 18:33:55.112331: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208381 140191300101952 af.py:224] 2022-01-20 18:33:55.112339: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.208422 140191300101952 af.py:224] 2022-01-20 18:33:55.112352: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208462 140191300101952 af.py:224] 2022-01-20 18:33:55.112361: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.208503 140191300101952 af.py:224] 2022-01-20 18:33:55.112377: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.208542 140191300101952 af.py:224] 2022-01-20 18:33:55.112390: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208583 140191300101952 af.py:224] 2022-01-20 18:33:55.112400: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208627 140191300101952 af.py:224] 2022-01-20 18:33:55.112410: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.208668 140191300101952 af.py:224] 2022-01-20 18:33:55.112422: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208709 140191300101952 af.py:224] 2022-01-20 18:33:55.112431: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.208749 140191300101952 af.py:224] 2022-01-20 18:33:55.112446: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.208805 140191300101952 af.py:224] 2022-01-20 18:33:55.112458: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208848 140191300101952 af.py:224] 2022-01-20 18:33:55.112468: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208889 140191300101952 af.py:224] 2022-01-20 18:33:55.112477: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.208947 140191300101952 af.py:224] 2022-01-20 18:33:55.112489: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.208989 140191300101952 af.py:224] 2022-01-20 18:33:55.112499: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.209030 140191300101952 af.py:224] 2022-01-20 18:33:55.112513: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.209085 140191300101952 af.py:224] 2022-01-20 18:33:55.112526: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209126 140191300101952 af.py:224] 2022-01-20 18:33:55.112536: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209166 140191300101952 af.py:224] 2022-01-20 18:33:55.112545: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.209207 140191300101952 af.py:224] 2022-01-20 18:33:55.112557: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209247 140191300101952 af.py:224] 2022-01-20 18:33:55.112565: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.209288 140191300101952 af.py:224] 2022-01-20 18:33:55.112578: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.209328 140191300101952 af.py:224] 2022-01-20 18:33:55.112607: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209372 140191300101952 af.py:224] 2022-01-20 18:33:55.112616: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209413 140191300101952 af.py:224] 2022-01-20 18:33:55.112622: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.209453 140191300101952 af.py:224] 2022-01-20 18:33:55.112634: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209494 140191300101952 af.py:224] 2022-01-20 18:33:55.112644: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.209535 140191300101952 af.py:224] 2022-01-20 18:33:55.112658: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.209576 140191300101952 af.py:224] 2022-01-20 18:33:55.112672: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209617 140191300101952 af.py:224] 2022-01-20 18:33:55.112683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209657 140191300101952 af.py:224] 2022-01-20 18:33:55.112691: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.209697 140191300101952 af.py:224] 2022-01-20 18:33:55.112703: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209738 140191300101952 af.py:224] 2022-01-20 18:33:55.112711: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.209778 140191300101952 af.py:224] 2022-01-20 18:33:55.112724: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.209819 140191300101952 af.py:224] 2022-01-20 18:33:55.112738: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209859 140191300101952 af.py:224] 2022-01-20 18:33:55.112749: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209900 140191300101952 af.py:224] 2022-01-20 18:33:55.112758: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.209941 140191300101952 af.py:224] 2022-01-20 18:33:55.112771: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.209982 140191300101952 af.py:224] 2022-01-20 18:33:55.112780: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.210023 140191300101952 af.py:224] 2022-01-20 18:33:55.112794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.210067 140191300101952 af.py:224] 2022-01-20 18:33:55.112808: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210108 140191300101952 af.py:224] 2022-01-20 18:33:55.112818: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210149 140191300101952 af.py:224] 2022-01-20 18:33:55.112828: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.210189 140191300101952 af.py:224] 2022-01-20 18:33:55.112839: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210230 140191300101952 af.py:224] 2022-01-20 18:33:55.112849: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.210271 140191300101952 af.py:224] 2022-01-20 18:33:55.112864: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.210312 140191300101952 af.py:224] 2022-01-20 18:33:55.112876: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210353 140191300101952 af.py:224] 2022-01-20 18:33:55.112886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210394 140191300101952 af.py:224] 2022-01-20 18:33:55.112894: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.210448 140191300101952 af.py:224] 2022-01-20 18:33:55.112906: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210490 140191300101952 af.py:224] 2022-01-20 18:33:55.112913: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.210530 140191300101952 af.py:224] 2022-01-20 18:33:55.112927: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.210571 140191300101952 af.py:224] 2022-01-20 18:33:55.112940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210612 140191300101952 af.py:224] 2022-01-20 18:33:55.112952: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210652 140191300101952 af.py:224] 2022-01-20 18:33:55.112962: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.210693 140191300101952 af.py:224] 2022-01-20 18:33:55.112974: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210734 140191300101952 af.py:224] 2022-01-20 18:33:55.112984: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.210779 140191300101952 af.py:224] 2022-01-20 18:33:55.112994: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.210820 140191300101952 af.py:224] 2022-01-20 18:33:55.113003: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210861 140191300101952 af.py:224] 2022-01-20 18:33:55.113009: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210902 140191300101952 af.py:224] 2022-01-20 18:33:55.113014: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.210943 140191300101952 af.py:224] 2022-01-20 18:33:55.113021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.210985 140191300101952 af.py:224] 2022-01-20 18:33:55.113026: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.211026 140191300101952 af.py:224] 2022-01-20 18:33:55.113034: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.211067 140191300101952 af.py:224] 2022-01-20 18:33:55.113041: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211107 140191300101952 af.py:224] 2022-01-20 18:33:55.113047: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211148 140191300101952 af.py:224] 2022-01-20 18:33:55.113052: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.211189 140191300101952 af.py:224] 2022-01-20 18:33:55.113059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211230 140191300101952 af.py:224] 2022-01-20 18:33:55.113064: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.211270 140191300101952 af.py:224] 2022-01-20 18:33:55.113072: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.211311 140191300101952 af.py:224] 2022-01-20 18:33:55.113079: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211351 140191300101952 af.py:224] 2022-01-20 18:33:55.113085: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211391 140191300101952 af.py:224] 2022-01-20 18:33:55.113090: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.211432 140191300101952 af.py:224] 2022-01-20 18:33:55.113097: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211476 140191300101952 af.py:224] 2022-01-20 18:33:55.113102: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.211517 140191300101952 af.py:224] 2022-01-20 18:33:55.113110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.211557 140191300101952 af.py:224] 2022-01-20 18:33:55.113117: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211597 140191300101952 af.py:224] 2022-01-20 18:33:55.113124: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211638 140191300101952 af.py:224] 2022-01-20 18:33:55.113129: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.211679 140191300101952 af.py:224] 2022-01-20 18:33:55.113136: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211720 140191300101952 af.py:224] 2022-01-20 18:33:55.113141: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.211760 140191300101952 af.py:224] 2022-01-20 18:33:55.113149: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.211801 140191300101952 af.py:224] 2022-01-20 18:33:55.113156: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211842 140191300101952 af.py:224] 2022-01-20 18:33:55.113162: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211883 140191300101952 af.py:224] 2022-01-20 18:33:55.113167: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.211924 140191300101952 af.py:224] 2022-01-20 18:33:55.113173: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.211964 140191300101952 af.py:224] 2022-01-20 18:33:55.113178: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.212005 140191300101952 af.py:224] 2022-01-20 18:33:55.113186: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.212059 140191300101952 af.py:224] 2022-01-20 18:33:55.113195: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212101 140191300101952 af.py:224] 2022-01-20 18:33:55.113202: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212172 140191300101952 af.py:224] 2022-01-20 18:33:55.113207: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.212228 140191300101952 af.py:224] 2022-01-20 18:33:55.113213: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212269 140191300101952 af.py:224] 2022-01-20 18:33:55.113218: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.212310 140191300101952 af.py:224] 2022-01-20 18:33:55.113227: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.212351 140191300101952 af.py:224] 2022-01-20 18:33:55.113233: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212391 140191300101952 af.py:224] 2022-01-20 18:33:55.113240: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212432 140191300101952 af.py:224] 2022-01-20 18:33:55.113245: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.212473 140191300101952 af.py:224] 2022-01-20 18:33:55.113252: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212514 140191300101952 af.py:224] 2022-01-20 18:33:55.113257: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.212554 140191300101952 af.py:224] 2022-01-20 18:33:55.113265: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.212594 140191300101952 af.py:224] 2022-01-20 18:33:55.113272: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212635 140191300101952 af.py:224] 2022-01-20 18:33:55.113279: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212676 140191300101952 af.py:224] 2022-01-20 18:33:55.113284: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.212717 140191300101952 af.py:224] 2022-01-20 18:33:55.113290: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212758 140191300101952 af.py:224] 2022-01-20 18:33:55.113295: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.212799 140191300101952 af.py:224] 2022-01-20 18:33:55.113304: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.212840 140191300101952 af.py:224] 2022-01-20 18:33:55.113311: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212884 140191300101952 af.py:224] 2022-01-20 18:33:55.113317: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.212925 140191300101952 af.py:224] 2022-01-20 18:33:55.113323: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.212966 140191300101952 af.py:224] 2022-01-20 18:33:55.113329: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213006 140191300101952 af.py:224] 2022-01-20 18:33:55.113334: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.213047 140191300101952 af.py:224] 2022-01-20 18:33:55.113342: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.213088 140191300101952 af.py:224] 2022-01-20 18:33:55.113349: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213129 140191300101952 af.py:224] 2022-01-20 18:33:55.113355: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213170 140191300101952 af.py:224] 2022-01-20 18:33:55.113361: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.213211 140191300101952 af.py:224] 2022-01-20 18:33:55.113367: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213252 140191300101952 af.py:224] 2022-01-20 18:33:55.113372: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.213293 140191300101952 af.py:224] 2022-01-20 18:33:55.113381: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.213333 140191300101952 af.py:224] 2022-01-20 18:33:55.113387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213374 140191300101952 af.py:224] 2022-01-20 18:33:55.113393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213415 140191300101952 af.py:224] 2022-01-20 18:33:55.113399: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.213456 140191300101952 af.py:224] 2022-01-20 18:33:55.113405: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213496 140191300101952 af.py:224] 2022-01-20 18:33:55.113410: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.213537 140191300101952 af.py:224] 2022-01-20 18:33:55.113418: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.213580 140191300101952 af.py:224] 2022-01-20 18:33:55.113425: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213622 140191300101952 af.py:224] 2022-01-20 18:33:55.113431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213662 140191300101952 af.py:224] 2022-01-20 18:33:55.113436: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.213717 140191300101952 af.py:224] 2022-01-20 18:33:55.113443: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213759 140191300101952 af.py:224] 2022-01-20 18:33:55.113448: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.213800 140191300101952 af.py:224] 2022-01-20 18:33:55.113456: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.213841 140191300101952 af.py:224] 2022-01-20 18:33:55.113463: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213882 140191300101952 af.py:224] 2022-01-20 18:33:55.113469: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.213923 140191300101952 af.py:224] 2022-01-20 18:33:55.113474: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.213964 140191300101952 af.py:224] 2022-01-20 18:33:55.113481: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214005 140191300101952 af.py:224] 2022-01-20 18:33:55.113486: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.214046 140191300101952 af.py:224] 2022-01-20 18:33:55.113494: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.214087 140191300101952 af.py:224] 2022-01-20 18:33:55.113501: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214127 140191300101952 af.py:224] 2022-01-20 18:33:55.113507: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214168 140191300101952 af.py:224] 2022-01-20 18:33:55.113512: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.214209 140191300101952 af.py:224] 2022-01-20 18:33:55.113519: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214251 140191300101952 af.py:224] 2022-01-20 18:33:55.113524: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.214295 140191300101952 af.py:224] 2022-01-20 18:33:55.113532: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.214336 140191300101952 af.py:224] 2022-01-20 18:33:55.113539: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214377 140191300101952 af.py:224] 2022-01-20 18:33:55.113545: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214418 140191300101952 af.py:224] 2022-01-20 18:33:55.113550: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.214459 140191300101952 af.py:224] 2022-01-20 18:33:55.113557: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214500 140191300101952 af.py:224] 2022-01-20 18:33:55.113562: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.214540 140191300101952 af.py:224] 2022-01-20 18:33:55.113570: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.214581 140191300101952 af.py:224] 2022-01-20 18:33:55.113578: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214622 140191300101952 af.py:224] 2022-01-20 18:33:55.113584: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214663 140191300101952 af.py:224] 2022-01-20 18:33:55.113589: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.214704 140191300101952 af.py:224] 2022-01-20 18:33:55.113596: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214744 140191300101952 af.py:224] 2022-01-20 18:33:55.113602: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.214784 140191300101952 af.py:224] 2022-01-20 18:33:55.113610: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.214825 140191300101952 af.py:224] 2022-01-20 18:33:55.113616: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214865 140191300101952 af.py:224] 2022-01-20 18:33:55.113622: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214906 140191300101952 af.py:224] 2022-01-20 18:33:55.113628: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.214947 140191300101952 af.py:224] 2022-01-20 18:33:55.113634: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.214991 140191300101952 af.py:224] 2022-01-20 18:33:55.113639: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.215032 140191300101952 af.py:224] 2022-01-20 18:33:55.113647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.215072 140191300101952 af.py:224] 2022-01-20 18:33:55.113654: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215113 140191300101952 af.py:224] 2022-01-20 18:33:55.113660: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215154 140191300101952 af.py:224] 2022-01-20 18:33:55.113665: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.215195 140191300101952 af.py:224] 2022-01-20 18:33:55.113672: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215236 140191300101952 af.py:224] 2022-01-20 18:33:55.113677: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.215276 140191300101952 af.py:224] 2022-01-20 18:33:55.113685: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.215317 140191300101952 af.py:224] 2022-01-20 18:33:55.113691: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215375 140191300101952 af.py:224] 2022-01-20 18:33:55.113698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215436 140191300101952 af.py:224] 2022-01-20 18:33:55.113703: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.215493 140191300101952 af.py:224] 2022-01-20 18:33:55.113709: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215534 140191300101952 af.py:224] 2022-01-20 18:33:55.113714: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.215600 140191300101952 af.py:224] 2022-01-20 18:33:55.113722: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.215642 140191300101952 af.py:224] 2022-01-20 18:33:55.113729: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215697 140191300101952 af.py:224] 2022-01-20 18:33:55.113735: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215742 140191300101952 af.py:224] 2022-01-20 18:33:55.113740: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.215783 140191300101952 af.py:224] 2022-01-20 18:33:55.113747: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215824 140191300101952 af.py:224] 2022-01-20 18:33:55.113752: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.215864 140191300101952 af.py:224] 2022-01-20 18:33:55.113760: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.215906 140191300101952 af.py:224] 2022-01-20 18:33:55.113767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215946 140191300101952 af.py:224] 2022-01-20 18:33:55.113773: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.215987 140191300101952 af.py:224] 2022-01-20 18:33:55.113778: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.216027 140191300101952 af.py:224] 2022-01-20 18:33:55.113785: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216067 140191300101952 af.py:224] 2022-01-20 18:33:55.113790: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.216108 140191300101952 af.py:224] 2022-01-20 18:33:55.113798: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.216175 140191300101952 af.py:224] 2022-01-20 18:33:55.113809: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216230 140191300101952 af.py:224] 2022-01-20 18:33:55.113817: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216271 140191300101952 af.py:224] 2022-01-20 18:33:55.113825: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.216311 140191300101952 af.py:224] 2022-01-20 18:33:55.113838: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216352 140191300101952 af.py:224] 2022-01-20 18:33:55.113847: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.216392 140191300101952 af.py:224] 2022-01-20 18:33:55.113860: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.216432 140191300101952 af.py:224] 2022-01-20 18:33:55.113870: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216477 140191300101952 af.py:224] 2022-01-20 18:33:55.113877: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216518 140191300101952 af.py:224] 2022-01-20 18:33:55.113883: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.216559 140191300101952 af.py:224] 2022-01-20 18:33:55.113889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216600 140191300101952 af.py:224] 2022-01-20 18:33:55.113894: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.216641 140191300101952 af.py:224] 2022-01-20 18:33:55.113902: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.216682 140191300101952 af.py:224] 2022-01-20 18:33:55.113909: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216722 140191300101952 af.py:224] 2022-01-20 18:33:55.113915: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216763 140191300101952 af.py:224] 2022-01-20 18:33:55.113920: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.216803 140191300101952 af.py:224] 2022-01-20 18:33:55.113927: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216844 140191300101952 af.py:224] 2022-01-20 18:33:55.113932: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.216885 140191300101952 af.py:224] 2022-01-20 18:33:55.113939: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.216925 140191300101952 af.py:224] 2022-01-20 18:33:55.113946: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.216967 140191300101952 af.py:224] 2022-01-20 18:33:55.113952: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217007 140191300101952 af.py:224] 2022-01-20 18:33:55.113958: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.217049 140191300101952 af.py:224] 2022-01-20 18:33:55.113963: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217104 140191300101952 af.py:224] 2022-01-20 18:33:55.113968: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.217145 140191300101952 af.py:224] 2022-01-20 18:33:55.113976: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.217190 140191300101952 af.py:224] 2022-01-20 18:33:55.113984: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217232 140191300101952 af.py:224] 2022-01-20 18:33:55.113990: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217273 140191300101952 af.py:224] 2022-01-20 18:33:55.113995: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.217314 140191300101952 af.py:224] 2022-01-20 18:33:55.114002: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217355 140191300101952 af.py:224] 2022-01-20 18:33:55.114009: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.217396 140191300101952 af.py:224] 2022-01-20 18:33:55.114023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.217436 140191300101952 af.py:224] 2022-01-20 18:33:55.114037: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217477 140191300101952 af.py:224] 2022-01-20 18:33:55.114048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217518 140191300101952 af.py:224] 2022-01-20 18:33:55.114057: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.217558 140191300101952 af.py:224] 2022-01-20 18:33:55.114069: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217599 140191300101952 af.py:224] 2022-01-20 18:33:55.114079: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.217640 140191300101952 af.py:224] 2022-01-20 18:33:55.114094: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.217681 140191300101952 af.py:224] 2022-01-20 18:33:55.114107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217722 140191300101952 af.py:224] 2022-01-20 18:33:55.114118: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217762 140191300101952 af.py:224] 2022-01-20 18:33:55.114126: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.217803 140191300101952 af.py:224] 2022-01-20 18:33:55.114134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217844 140191300101952 af.py:224] 2022-01-20 18:33:55.114141: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.217889 140191300101952 af.py:224] 2022-01-20 18:33:55.114151: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.217930 140191300101952 af.py:224] 2022-01-20 18:33:55.114161: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.217970 140191300101952 af.py:224] 2022-01-20 18:33:55.114168: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218010 140191300101952 af.py:224] 2022-01-20 18:33:55.114173: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.218050 140191300101952 af.py:224] 2022-01-20 18:33:55.114181: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218091 140191300101952 af.py:224] 2022-01-20 18:33:55.114187: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.218131 140191300101952 af.py:224] 2022-01-20 18:33:55.114195: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.218172 140191300101952 af.py:224] 2022-01-20 18:33:55.114203: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218213 140191300101952 af.py:224] 2022-01-20 18:33:55.114209: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218254 140191300101952 af.py:224] 2022-01-20 18:33:55.114214: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.218294 140191300101952 af.py:224] 2022-01-20 18:33:55.114223: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218335 140191300101952 af.py:224] 2022-01-20 18:33:55.114229: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.218376 140191300101952 af.py:224] 2022-01-20 18:33:55.114237: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.218417 140191300101952 af.py:224] 2022-01-20 18:33:55.114244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218457 140191300101952 af.py:224] 2022-01-20 18:33:55.114251: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218498 140191300101952 af.py:224] 2022-01-20 18:33:55.114256: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.218542 140191300101952 af.py:224] 2022-01-20 18:33:55.114263: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218583 140191300101952 af.py:224] 2022-01-20 18:33:55.114268: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.218624 140191300101952 af.py:224] 2022-01-20 18:33:55.114276: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.218664 140191300101952 af.py:224] 2022-01-20 18:33:55.114283: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218719 140191300101952 af.py:224] 2022-01-20 18:33:55.114289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218760 140191300101952 af.py:224] 2022-01-20 18:33:55.114295: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.218801 140191300101952 af.py:224] 2022-01-20 18:33:55.114305: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218841 140191300101952 af.py:224] 2022-01-20 18:33:55.114311: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.218882 140191300101952 af.py:224] 2022-01-20 18:33:55.114335: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.218922 140191300101952 af.py:224] 2022-01-20 18:33:55.114344: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.218963 140191300101952 af.py:224] 2022-01-20 18:33:55.114350: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219003 140191300101952 af.py:224] 2022-01-20 18:33:55.114355: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.219043 140191300101952 af.py:224] 2022-01-20 18:33:55.114362: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219084 140191300101952 af.py:224] 2022-01-20 18:33:55.114367: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.219124 140191300101952 af.py:224] 2022-01-20 18:33:55.114375: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.219163 140191300101952 af.py:224] 2022-01-20 18:33:55.114381: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219204 140191300101952 af.py:224] 2022-01-20 18:33:55.114387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219248 140191300101952 af.py:224] 2022-01-20 18:33:55.114393: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.219289 140191300101952 af.py:224] 2022-01-20 18:33:55.114399: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219329 140191300101952 af.py:224] 2022-01-20 18:33:55.114404: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.219371 140191300101952 af.py:224] 2022-01-20 18:33:55.114412: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.219411 140191300101952 af.py:224] 2022-01-20 18:33:55.114418: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219451 140191300101952 af.py:224] 2022-01-20 18:33:55.114425: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219492 140191300101952 af.py:224] 2022-01-20 18:33:55.114430: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.219532 140191300101952 af.py:224] 2022-01-20 18:33:55.114436: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219573 140191300101952 af.py:224] 2022-01-20 18:33:55.114441: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.219614 140191300101952 af.py:224] 2022-01-20 18:33:55.114453: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.219654 140191300101952 af.py:224] 2022-01-20 18:33:55.114464: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219695 140191300101952 af.py:224] 2022-01-20 18:33:55.114471: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219736 140191300101952 af.py:224] 2022-01-20 18:33:55.114476: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.219777 140191300101952 af.py:224] 2022-01-20 18:33:55.114483: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219817 140191300101952 af.py:224] 2022-01-20 18:33:55.114488: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.219858 140191300101952 af.py:224] 2022-01-20 18:33:55.114496: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.219899 140191300101952 af.py:224] 2022-01-20 18:33:55.114503: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219943 140191300101952 af.py:224] 2022-01-20 18:33:55.114509: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.219984 140191300101952 af.py:224] 2022-01-20 18:33:55.114514: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.220024 140191300101952 af.py:224] 2022-01-20 18:33:55.114521: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220065 140191300101952 af.py:224] 2022-01-20 18:33:55.114526: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.220105 140191300101952 af.py:224] 2022-01-20 18:33:55.114536: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.220177 140191300101952 af.py:224] 2022-01-20 18:33:55.114544: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220232 140191300101952 af.py:224] 2022-01-20 18:33:55.114551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220273 140191300101952 af.py:224] 2022-01-20 18:33:55.114556: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.220313 140191300101952 af.py:224] 2022-01-20 18:33:55.114562: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220369 140191300101952 af.py:224] 2022-01-20 18:33:55.114567: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.220411 140191300101952 af.py:224] 2022-01-20 18:33:55.114574: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.220452 140191300101952 af.py:224] 2022-01-20 18:33:55.114581: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220493 140191300101952 af.py:224] 2022-01-20 18:33:55.114588: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220534 140191300101952 af.py:224] 2022-01-20 18:33:55.114595: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.220575 140191300101952 af.py:224] 2022-01-20 18:33:55.114606: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220615 140191300101952 af.py:224] 2022-01-20 18:33:55.114616: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.220655 140191300101952 af.py:224] 2022-01-20 18:33:55.114631: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.220700 140191300101952 af.py:224] 2022-01-20 18:33:55.114644: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220741 140191300101952 af.py:224] 2022-01-20 18:33:55.114655: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220782 140191300101952 af.py:224] 2022-01-20 18:33:55.114663: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.220823 140191300101952 af.py:224] 2022-01-20 18:33:55.114674: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220864 140191300101952 af.py:224] 2022-01-20 18:33:55.114683: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.220904 140191300101952 af.py:224] 2022-01-20 18:33:55.114699: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.220945 140191300101952 af.py:224] 2022-01-20 18:33:55.114711: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.220985 140191300101952 af.py:224] 2022-01-20 18:33:55.114722: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221026 140191300101952 af.py:224] 2022-01-20 18:33:55.114731: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.221066 140191300101952 af.py:224] 2022-01-20 18:33:55.114744: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221107 140191300101952 af.py:224] 2022-01-20 18:33:55.114754: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.221148 140191300101952 af.py:224] 2022-01-20 18:33:55.114768: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.221189 140191300101952 af.py:224] 2022-01-20 18:33:55.114781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221230 140191300101952 af.py:224] 2022-01-20 18:33:55.114793: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221271 140191300101952 af.py:224] 2022-01-20 18:33:55.114802: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.221311 140191300101952 af.py:224] 2022-01-20 18:33:55.114810: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221358 140191300101952 af.py:224] 2022-01-20 18:33:55.114815: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.221399 140191300101952 af.py:224] 2022-01-20 18:33:55.114825: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.221439 140191300101952 af.py:224] 2022-01-20 18:33:55.114833: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221479 140191300101952 af.py:224] 2022-01-20 18:33:55.114839: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221520 140191300101952 af.py:224] 2022-01-20 18:33:55.114844: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.221560 140191300101952 af.py:224] 2022-01-20 18:33:55.114851: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221601 140191300101952 af.py:224] 2022-01-20 18:33:55.114859: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.221641 140191300101952 af.py:224] 2022-01-20 18:33:55.114873: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.221682 140191300101952 af.py:224] 2022-01-20 18:33:55.114886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221722 140191300101952 af.py:224] 2022-01-20 18:33:55.114897: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221763 140191300101952 af.py:224] 2022-01-20 18:33:55.114905: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.221803 140191300101952 af.py:224] 2022-01-20 18:33:55.114917: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221844 140191300101952 af.py:224] 2022-01-20 18:33:55.114925: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.221885 140191300101952 af.py:224] 2022-01-20 18:33:55.114940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.221925 140191300101952 af.py:224] 2022-01-20 18:33:55.114951: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.221985 140191300101952 af.py:224] 2022-01-20 18:33:55.114961: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222028 140191300101952 af.py:224] 2022-01-20 18:33:55.114970: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.222098 140191300101952 af.py:224] 2022-01-20 18:33:55.114983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222140 140191300101952 af.py:224] 2022-01-20 18:33:55.114992: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.222180 140191300101952 af.py:224] 2022-01-20 18:33:55.115007: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.222222 140191300101952 af.py:224] 2022-01-20 18:33:55.115021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222277 140191300101952 af.py:224] 2022-01-20 18:33:55.115032: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222318 140191300101952 af.py:224] 2022-01-20 18:33:55.115038: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.222359 140191300101952 af.py:224] 2022-01-20 18:33:55.115050: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222399 140191300101952 af.py:224] 2022-01-20 18:33:55.115059: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.222440 140191300101952 af.py:224] 2022-01-20 18:33:55.115075: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.222481 140191300101952 af.py:224] 2022-01-20 18:33:55.115089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222522 140191300101952 af.py:224] 2022-01-20 18:33:55.115098: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222563 140191300101952 af.py:224] 2022-01-20 18:33:55.115107: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.222604 140191300101952 af.py:224] 2022-01-20 18:33:55.115119: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222645 140191300101952 af.py:224] 2022-01-20 18:33:55.115128: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.222685 140191300101952 af.py:224] 2022-01-20 18:33:55.115138: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.222725 140191300101952 af.py:224] 2022-01-20 18:33:55.115148: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222766 140191300101952 af.py:224] 2022-01-20 18:33:55.115158: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222809 140191300101952 af.py:224] 2022-01-20 18:33:55.115168: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.222850 140191300101952 af.py:224] 2022-01-20 18:33:55.115179: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.222891 140191300101952 af.py:224] 2022-01-20 18:33:55.115188: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.222931 140191300101952 af.py:224] 2022-01-20 18:33:55.115203: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.222972 140191300101952 af.py:224] 2022-01-20 18:33:55.115216: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223013 140191300101952 af.py:224] 2022-01-20 18:33:55.115227: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223053 140191300101952 af.py:224] 2022-01-20 18:33:55.115237: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.223093 140191300101952 af.py:224] 2022-01-20 18:33:55.115249: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223134 140191300101952 af.py:224] 2022-01-20 18:33:55.115256: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.223174 140191300101952 af.py:224] 2022-01-20 18:33:55.115270: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.223215 140191300101952 af.py:224] 2022-01-20 18:33:55.115283: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223256 140191300101952 af.py:224] 2022-01-20 18:33:55.115294: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223297 140191300101952 af.py:224] 2022-01-20 18:33:55.115303: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.223336 140191300101952 af.py:224] 2022-01-20 18:33:55.115315: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223377 140191300101952 af.py:224] 2022-01-20 18:33:55.115324: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.223418 140191300101952 af.py:224] 2022-01-20 18:33:55.115339: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.223459 140191300101952 af.py:224] 2022-01-20 18:33:55.115352: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223502 140191300101952 af.py:224] 2022-01-20 18:33:55.115363: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223543 140191300101952 af.py:224] 2022-01-20 18:33:55.115370: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.223584 140191300101952 af.py:224] 2022-01-20 18:33:55.115382: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223625 140191300101952 af.py:224] 2022-01-20 18:33:55.115391: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.223679 140191300101952 af.py:224] 2022-01-20 18:33:55.115407: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.223721 140191300101952 af.py:224] 2022-01-20 18:33:55.115419: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223761 140191300101952 af.py:224] 2022-01-20 18:33:55.115426: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223802 140191300101952 af.py:224] 2022-01-20 18:33:55.115432: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.223843 140191300101952 af.py:224] 2022-01-20 18:33:55.115438: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.223883 140191300101952 af.py:224] 2022-01-20 18:33:55.115443: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.223923 140191300101952 af.py:224] 2022-01-20 18:33:55.115452: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.223965 140191300101952 af.py:224] 2022-01-20 18:33:55.115459: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224006 140191300101952 af.py:224] 2022-01-20 18:33:55.115467: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224047 140191300101952 af.py:224] 2022-01-20 18:33:55.115476: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.224087 140191300101952 af.py:224] 2022-01-20 18:33:55.115486: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224131 140191300101952 af.py:224] 2022-01-20 18:33:55.115493: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.224215 140191300101952 af.py:224] 2022-01-20 18:33:55.115502: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.224259 140191300101952 af.py:224] 2022-01-20 18:33:55.115511: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224300 140191300101952 af.py:224] 2022-01-20 18:33:55.115518: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224341 140191300101952 af.py:224] 2022-01-20 18:33:55.115523: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.224381 140191300101952 af.py:224] 2022-01-20 18:33:55.115530: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224423 140191300101952 af.py:224] 2022-01-20 18:33:55.115535: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.224464 140191300101952 af.py:224] 2022-01-20 18:33:55.115543: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.224504 140191300101952 af.py:224] 2022-01-20 18:33:55.115551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224545 140191300101952 af.py:224] 2022-01-20 18:33:55.115557: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224586 140191300101952 af.py:224] 2022-01-20 18:33:55.115563: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.224627 140191300101952 af.py:224] 2022-01-20 18:33:55.115570: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224667 140191300101952 af.py:224] 2022-01-20 18:33:55.115575: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.224708 140191300101952 af.py:224] 2022-01-20 18:33:55.115583: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.224749 140191300101952 af.py:224] 2022-01-20 18:33:55.115591: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224789 140191300101952 af.py:224] 2022-01-20 18:33:55.115597: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224830 140191300101952 af.py:224] 2022-01-20 18:33:55.115602: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.224870 140191300101952 af.py:224] 2022-01-20 18:33:55.115609: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.224915 140191300101952 af.py:224] 2022-01-20 18:33:55.115614: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.224956 140191300101952 af.py:224] 2022-01-20 18:33:55.115627: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.224997 140191300101952 af.py:224] 2022-01-20 18:33:55.115639: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225038 140191300101952 af.py:224] 2022-01-20 18:33:55.115647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225078 140191300101952 af.py:224] 2022-01-20 18:33:55.115653: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.225119 140191300101952 af.py:224] 2022-01-20 18:33:55.115660: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225160 140191300101952 af.py:224] 2022-01-20 18:33:55.115665: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.225200 140191300101952 af.py:224] 2022-01-20 18:33:55.115672: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.225241 140191300101952 af.py:224] 2022-01-20 18:33:55.115680: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225282 140191300101952 af.py:224] 2022-01-20 18:33:55.115686: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225337 140191300101952 af.py:224] 2022-01-20 18:33:55.115692: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.225378 140191300101952 af.py:224] 2022-01-20 18:33:55.115704: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225420 140191300101952 af.py:224] 2022-01-20 18:33:55.115713: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.225461 140191300101952 af.py:224] 2022-01-20 18:33:55.115723: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.225501 140191300101952 af.py:224] 2022-01-20 18:33:55.115731: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225542 140191300101952 af.py:224] 2022-01-20 18:33:55.115753: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225582 140191300101952 af.py:224] 2022-01-20 18:33:55.115759: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.225626 140191300101952 af.py:224] 2022-01-20 18:33:55.115766: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225667 140191300101952 af.py:224] 2022-01-20 18:33:55.115774: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.225707 140191300101952 af.py:224] 2022-01-20 18:33:55.115783: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.225748 140191300101952 af.py:224] 2022-01-20 18:33:55.115791: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225788 140191300101952 af.py:224] 2022-01-20 18:33:55.115797: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225829 140191300101952 af.py:224] 2022-01-20 18:33:55.115802: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.225870 140191300101952 af.py:224] 2022-01-20 18:33:55.115809: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.225911 140191300101952 af.py:224] 2022-01-20 18:33:55.115814: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.225951 140191300101952 af.py:224] 2022-01-20 18:33:55.115823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.225992 140191300101952 af.py:224] 2022-01-20 18:33:55.115834: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226033 140191300101952 af.py:224] 2022-01-20 18:33:55.115845: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226073 140191300101952 af.py:224] 2022-01-20 18:33:55.115853: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.226114 140191300101952 af.py:224] 2022-01-20 18:33:55.115861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226154 140191300101952 af.py:224] 2022-01-20 18:33:55.115866: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.226194 140191300101952 af.py:224] 2022-01-20 18:33:55.115875: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.226235 140191300101952 af.py:224] 2022-01-20 18:33:55.115883: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226276 140191300101952 af.py:224] 2022-01-20 18:33:55.115890: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226320 140191300101952 af.py:224] 2022-01-20 18:33:55.115895: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.226361 140191300101952 af.py:224] 2022-01-20 18:33:55.115902: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226402 140191300101952 af.py:224] 2022-01-20 18:33:55.115907: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.226443 140191300101952 af.py:224] 2022-01-20 18:33:55.115915: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.226483 140191300101952 af.py:224] 2022-01-20 18:33:55.115922: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226524 140191300101952 af.py:224] 2022-01-20 18:33:55.115928: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226564 140191300101952 af.py:224] 2022-01-20 18:33:55.115933: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.226605 140191300101952 af.py:224] 2022-01-20 18:33:55.115940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226646 140191300101952 af.py:224] 2022-01-20 18:33:55.115945: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.226687 140191300101952 af.py:224] 2022-01-20 18:33:55.115954: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.226728 140191300101952 af.py:224] 2022-01-20 18:33:55.115960: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226768 140191300101952 af.py:224] 2022-01-20 18:33:55.115966: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226809 140191300101952 af.py:224] 2022-01-20 18:33:55.115971: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.226849 140191300101952 af.py:224] 2022-01-20 18:33:55.115978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.226890 140191300101952 af.py:224] 2022-01-20 18:33:55.115984: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.226943 140191300101952 af.py:224] 2022-01-20 18:33:55.115992: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.226985 140191300101952 af.py:224] 2022-01-20 18:33:55.115999: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227030 140191300101952 af.py:224] 2022-01-20 18:33:55.116005: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227071 140191300101952 af.py:224] 2022-01-20 18:33:55.116010: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.227112 140191300101952 af.py:224] 2022-01-20 18:33:55.116017: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227152 140191300101952 af.py:224] 2022-01-20 18:33:55.116022: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.227193 140191300101952 af.py:224] 2022-01-20 18:33:55.116030: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.227234 140191300101952 af.py:224] 2022-01-20 18:33:55.116038: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227275 140191300101952 af.py:224] 2022-01-20 18:33:55.116044: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227316 140191300101952 af.py:224] 2022-01-20 18:33:55.116050: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.227357 140191300101952 af.py:224] 2022-01-20 18:33:55.116057: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227398 140191300101952 af.py:224] 2022-01-20 18:33:55.116062: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.227439 140191300101952 af.py:224] 2022-01-20 18:33:55.116071: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.227479 140191300101952 af.py:224] 2022-01-20 18:33:55.116081: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227519 140191300101952 af.py:224] 2022-01-20 18:33:55.116091: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227560 140191300101952 af.py:224] 2022-01-20 18:33:55.116101: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.227600 140191300101952 af.py:224] 2022-01-20 18:33:55.116113: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227640 140191300101952 af.py:224] 2022-01-20 18:33:55.116128: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.227681 140191300101952 af.py:224] 2022-01-20 18:33:55.116140: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.227725 140191300101952 af.py:224] 2022-01-20 18:33:55.116148: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227766 140191300101952 af.py:224] 2022-01-20 18:33:55.116154: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227806 140191300101952 af.py:224] 2022-01-20 18:33:55.116160: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.227846 140191300101952 af.py:224] 2022-01-20 18:33:55.116167: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.227887 140191300101952 af.py:224] 2022-01-20 18:33:55.116172: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.227928 140191300101952 af.py:224] 2022-01-20 18:33:55.116179: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.227968 140191300101952 af.py:224] 2022-01-20 18:33:55.116189: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228009 140191300101952 af.py:224] 2022-01-20 18:33:55.116198: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228049 140191300101952 af.py:224] 2022-01-20 18:33:55.116206: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.228089 140191300101952 af.py:224] 2022-01-20 18:33:55.116219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228133 140191300101952 af.py:224] 2022-01-20 18:33:55.116227: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.228222 140191300101952 af.py:224] 2022-01-20 18:33:55.116240: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.228263 140191300101952 af.py:224] 2022-01-20 18:33:55.116253: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228303 140191300101952 af.py:224] 2022-01-20 18:33:55.116263: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228343 140191300101952 af.py:224] 2022-01-20 18:33:55.116272: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.228384 140191300101952 af.py:224] 2022-01-20 18:33:55.116284: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228428 140191300101952 af.py:224] 2022-01-20 18:33:55.116293: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.228469 140191300101952 af.py:224] 2022-01-20 18:33:55.116307: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.228510 140191300101952 af.py:224] 2022-01-20 18:33:55.116319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228550 140191300101952 af.py:224] 2022-01-20 18:33:55.116329: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228606 140191300101952 af.py:224] 2022-01-20 18:33:55.116338: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.228650 140191300101952 af.py:224] 2022-01-20 18:33:55.116350: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228690 140191300101952 af.py:224] 2022-01-20 18:33:55.116358: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.228765 140191300101952 af.py:224] 2022-01-20 18:33:55.116388: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.228823 140191300101952 af.py:224] 2022-01-20 18:33:55.116400: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228864 140191300101952 af.py:224] 2022-01-20 18:33:55.116409: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.228921 140191300101952 af.py:224] 2022-01-20 18:33:55.116418: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.228962 140191300101952 af.py:224] 2022-01-20 18:33:55.116430: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229002 140191300101952 af.py:224] 2022-01-20 18:33:55.116439: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.229043 140191300101952 af.py:224] 2022-01-20 18:33:55.116452: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.229084 140191300101952 af.py:224] 2022-01-20 18:33:55.116465: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229124 140191300101952 af.py:224] 2022-01-20 18:33:55.116475: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229165 140191300101952 af.py:224] 2022-01-20 18:33:55.116480: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.229209 140191300101952 af.py:224] 2022-01-20 18:33:55.116488: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229250 140191300101952 af.py:224] 2022-01-20 18:33:55.116493: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.229291 140191300101952 af.py:224] 2022-01-20 18:33:55.116501: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.229331 140191300101952 af.py:224] 2022-01-20 18:33:55.116510: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229372 140191300101952 af.py:224] 2022-01-20 18:33:55.116516: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229413 140191300101952 af.py:224] 2022-01-20 18:33:55.116522: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.229454 140191300101952 af.py:224] 2022-01-20 18:33:55.116533: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229494 140191300101952 af.py:224] 2022-01-20 18:33:55.116541: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.229535 140191300101952 af.py:224] 2022-01-20 18:33:55.116556: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.229575 140191300101952 af.py:224] 2022-01-20 18:33:55.116565: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229616 140191300101952 af.py:224] 2022-01-20 18:33:55.116572: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229657 140191300101952 af.py:224] 2022-01-20 18:33:55.116577: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.229697 140191300101952 af.py:224] 2022-01-20 18:33:55.116584: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229738 140191300101952 af.py:224] 2022-01-20 18:33:55.116589: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.229779 140191300101952 af.py:224] 2022-01-20 18:33:55.116597: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.229819 140191300101952 af.py:224] 2022-01-20 18:33:55.116607: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229860 140191300101952 af.py:224] 2022-01-20 18:33:55.116618: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229904 140191300101952 af.py:224] 2022-01-20 18:33:55.116627: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.229945 140191300101952 af.py:224] 2022-01-20 18:33:55.116638: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.229986 140191300101952 af.py:224] 2022-01-20 18:33:55.116647: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.230027 140191300101952 af.py:224] 2022-01-20 18:33:55.116662: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.230068 140191300101952 af.py:224] 2022-01-20 18:33:55.116676: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230109 140191300101952 af.py:224] 2022-01-20 18:33:55.116687: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230150 140191300101952 af.py:224] 2022-01-20 18:33:55.116696: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.230190 140191300101952 af.py:224] 2022-01-20 18:33:55.116708: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230231 140191300101952 af.py:224] 2022-01-20 18:33:55.116717: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.230286 140191300101952 af.py:224] 2022-01-20 18:33:55.116731: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.230327 140191300101952 af.py:224] 2022-01-20 18:33:55.116745: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230368 140191300101952 af.py:224] 2022-01-20 18:33:55.116757: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230409 140191300101952 af.py:224] 2022-01-20 18:33:55.116767: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.230449 140191300101952 af.py:224] 2022-01-20 18:33:55.116780: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230490 140191300101952 af.py:224] 2022-01-20 18:33:55.116789: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.230531 140191300101952 af.py:224] 2022-01-20 18:33:55.116805: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.230571 140191300101952 af.py:224] 2022-01-20 18:33:55.116818: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230615 140191300101952 af.py:224] 2022-01-20 18:33:55.116829: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230656 140191300101952 af.py:224] 2022-01-20 18:33:55.116839: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.230696 140191300101952 af.py:224] 2022-01-20 18:33:55.116853: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230736 140191300101952 af.py:224] 2022-01-20 18:33:55.116862: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.230777 140191300101952 af.py:224] 2022-01-20 18:33:55.116877: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.230818 140191300101952 af.py:224] 2022-01-20 18:33:55.116889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230858 140191300101952 af.py:224] 2022-01-20 18:33:55.116897: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230899 140191300101952 af.py:224] 2022-01-20 18:33:55.116903: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.230940 140191300101952 af.py:224] 2022-01-20 18:33:55.116912: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.230980 140191300101952 af.py:224] 2022-01-20 18:33:55.116922: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.231021 140191300101952 af.py:224] 2022-01-20 18:33:55.116937: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.231061 140191300101952 af.py:224] 2022-01-20 18:33:55.116948: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231101 140191300101952 af.py:224] 2022-01-20 18:33:55.116955: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231141 140191300101952 af.py:224] 2022-01-20 18:33:55.116960: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.231182 140191300101952 af.py:224] 2022-01-20 18:33:55.116967: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231222 140191300101952 af.py:224] 2022-01-20 18:33:55.116973: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.231262 140191300101952 af.py:224] 2022-01-20 18:33:55.116981: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.231306 140191300101952 af.py:224] 2022-01-20 18:33:55.116988: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231346 140191300101952 af.py:224] 2022-01-20 18:33:55.116997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231388 140191300101952 af.py:224] 2022-01-20 18:33:55.117002: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.231428 140191300101952 af.py:224] 2022-01-20 18:33:55.117009: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231468 140191300101952 af.py:224] 2022-01-20 18:33:55.117014: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.231509 140191300101952 af.py:224] 2022-01-20 18:33:55.117023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.231550 140191300101952 af.py:224] 2022-01-20 18:33:55.117030: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231591 140191300101952 af.py:224] 2022-01-20 18:33:55.117036: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231631 140191300101952 af.py:224] 2022-01-20 18:33:55.117041: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.231672 140191300101952 af.py:224] 2022-01-20 18:33:55.117049: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231713 140191300101952 af.py:224] 2022-01-20 18:33:55.117058: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.231753 140191300101952 af.py:224] 2022-01-20 18:33:55.117073: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.231793 140191300101952 af.py:224] 2022-01-20 18:33:55.117086: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231834 140191300101952 af.py:224] 2022-01-20 18:33:55.117097: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231874 140191300101952 af.py:224] 2022-01-20 18:33:55.117106: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.231928 140191300101952 af.py:224] 2022-01-20 18:33:55.117118: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.231973 140191300101952 af.py:224] 2022-01-20 18:33:55.117128: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.232014 140191300101952 af.py:224] 2022-01-20 18:33:55.117144: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.232054 140191300101952 af.py:224] 2022-01-20 18:33:55.117157: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232096 140191300101952 af.py:224] 2022-01-20 18:33:55.117165: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232140 140191300101952 af.py:224] 2022-01-20 18:33:55.117170: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.232181 140191300101952 af.py:224] 2022-01-20 18:33:55.117177: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232222 140191300101952 af.py:224] 2022-01-20 18:33:55.117183: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.232262 140191300101952 af.py:224] 2022-01-20 18:33:55.117191: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.232303 140191300101952 af.py:224] 2022-01-20 18:33:55.117199: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232343 140191300101952 af.py:224] 2022-01-20 18:33:55.117206: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232384 140191300101952 af.py:224] 2022-01-20 18:33:55.117211: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.232424 140191300101952 af.py:224] 2022-01-20 18:33:55.117217: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232465 140191300101952 af.py:224] 2022-01-20 18:33:55.117222: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.232506 140191300101952 af.py:224] 2022-01-20 18:33:55.117230: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.232547 140191300101952 af.py:224] 2022-01-20 18:33:55.117241: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232587 140191300101952 af.py:224] 2022-01-20 18:33:55.117252: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232628 140191300101952 af.py:224] 2022-01-20 18:33:55.117261: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.232672 140191300101952 af.py:224] 2022-01-20 18:33:55.117273: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232713 140191300101952 af.py:224] 2022-01-20 18:33:55.117282: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.232754 140191300101952 af.py:224] 2022-01-20 18:33:55.117295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.232794 140191300101952 af.py:224] 2022-01-20 18:33:55.117307: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232835 140191300101952 af.py:224] 2022-01-20 18:33:55.117319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232876 140191300101952 af.py:224] 2022-01-20 18:33:55.117328: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.232916 140191300101952 af.py:224] 2022-01-20 18:33:55.117341: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.232956 140191300101952 af.py:224] 2022-01-20 18:33:55.117351: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.232997 140191300101952 af.py:224] 2022-01-20 18:33:55.117366: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.233038 140191300101952 af.py:224] 2022-01-20 18:33:55.117379: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233078 140191300101952 af.py:224] 2022-01-20 18:33:55.117389: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233119 140191300101952 af.py:224] 2022-01-20 18:33:55.117399: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.233159 140191300101952 af.py:224] 2022-01-20 18:33:55.117411: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233199 140191300101952 af.py:224] 2022-01-20 18:33:55.117420: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.233240 140191300101952 af.py:224] 2022-01-20 18:33:55.117434: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.233280 140191300101952 af.py:224] 2022-01-20 18:33:55.117446: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233321 140191300101952 af.py:224] 2022-01-20 18:33:55.117457: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233365 140191300101952 af.py:224] 2022-01-20 18:33:55.117466: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.233407 140191300101952 af.py:224] 2022-01-20 18:33:55.117478: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233447 140191300101952 af.py:224] 2022-01-20 18:33:55.117488: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.233488 140191300101952 af.py:224] 2022-01-20 18:33:55.117498: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.233542 140191300101952 af.py:224] 2022-01-20 18:33:55.117507: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233584 140191300101952 af.py:224] 2022-01-20 18:33:55.117514: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233626 140191300101952 af.py:224] 2022-01-20 18:33:55.117519: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.233667 140191300101952 af.py:224] 2022-01-20 18:33:55.117526: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233708 140191300101952 af.py:224] 2022-01-20 18:33:55.117532: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.233749 140191300101952 af.py:224] 2022-01-20 18:33:55.117541: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.233789 140191300101952 af.py:224] 2022-01-20 18:33:55.117551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233831 140191300101952 af.py:224] 2022-01-20 18:33:55.117561: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233871 140191300101952 af.py:224] 2022-01-20 18:33:55.117570: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.233912 140191300101952 af.py:224] 2022-01-20 18:33:55.117582: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.233952 140191300101952 af.py:224] 2022-01-20 18:33:55.117592: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.233993 140191300101952 af.py:224] 2022-01-20 18:33:55.117604: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.234033 140191300101952 af.py:224] 2022-01-20 18:33:55.117612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234078 140191300101952 af.py:224] 2022-01-20 18:33:55.117618: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234119 140191300101952 af.py:224] 2022-01-20 18:33:55.117624: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.234159 140191300101952 af.py:224] 2022-01-20 18:33:55.117631: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234200 140191300101952 af.py:224] 2022-01-20 18:33:55.117636: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.234240 140191300101952 af.py:224] 2022-01-20 18:33:55.117644: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.234280 140191300101952 af.py:224] 2022-01-20 18:33:55.117654: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234321 140191300101952 af.py:224] 2022-01-20 18:33:55.117663: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234361 140191300101952 af.py:224] 2022-01-20 18:33:55.117671: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.234402 140191300101952 af.py:224] 2022-01-20 18:33:55.117683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234443 140191300101952 af.py:224] 2022-01-20 18:33:55.117691: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.234483 140191300101952 af.py:224] 2022-01-20 18:33:55.117703: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.234524 140191300101952 af.py:224] 2022-01-20 18:33:55.117716: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234565 140191300101952 af.py:224] 2022-01-20 18:33:55.117727: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234606 140191300101952 af.py:224] 2022-01-20 18:33:55.117737: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.234646 140191300101952 af.py:224] 2022-01-20 18:33:55.117750: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234687 140191300101952 af.py:224] 2022-01-20 18:33:55.117759: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.234728 140191300101952 af.py:224] 2022-01-20 18:33:55.117769: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.234772 140191300101952 af.py:224] 2022-01-20 18:33:55.117778: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234813 140191300101952 af.py:224] 2022-01-20 18:33:55.117784: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234854 140191300101952 af.py:224] 2022-01-20 18:33:55.117789: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.234895 140191300101952 af.py:224] 2022-01-20 18:33:55.117796: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.234936 140191300101952 af.py:224] 2022-01-20 18:33:55.117801: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.234977 140191300101952 af.py:224] 2022-01-20 18:33:55.117811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.235017 140191300101952 af.py:224] 2022-01-20 18:33:55.117821: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235058 140191300101952 af.py:224] 2022-01-20 18:33:55.117832: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235099 140191300101952 af.py:224] 2022-01-20 18:33:55.117841: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.235156 140191300101952 af.py:224] 2022-01-20 18:33:55.117851: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235199 140191300101952 af.py:224] 2022-01-20 18:33:55.117861: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.235240 140191300101952 af.py:224] 2022-01-20 18:33:55.117874: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.235314 140191300101952 af.py:224] 2022-01-20 18:33:55.117883: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235372 140191300101952 af.py:224] 2022-01-20 18:33:55.117889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235414 140191300101952 af.py:224] 2022-01-20 18:33:55.117895: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.235470 140191300101952 af.py:224] 2022-01-20 18:33:55.117901: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235514 140191300101952 af.py:224] 2022-01-20 18:33:55.117907: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.235555 140191300101952 af.py:224] 2022-01-20 18:33:55.117915: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.235596 140191300101952 af.py:224] 2022-01-20 18:33:55.117922: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235637 140191300101952 af.py:224] 2022-01-20 18:33:55.117931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235678 140191300101952 af.py:224] 2022-01-20 18:33:55.117940: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.235719 140191300101952 af.py:224] 2022-01-20 18:33:55.117949: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235760 140191300101952 af.py:224] 2022-01-20 18:33:55.117954: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.235800 140191300101952 af.py:224] 2022-01-20 18:33:55.117962: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.235840 140191300101952 af.py:224] 2022-01-20 18:33:55.117970: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235881 140191300101952 af.py:224] 2022-01-20 18:33:55.117977: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.235921 140191300101952 af.py:224] 2022-01-20 18:33:55.117984: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.235962 140191300101952 af.py:224] 2022-01-20 18:33:55.117995: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236003 140191300101952 af.py:224] 2022-01-20 18:33:55.118005: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.236044 140191300101952 af.py:224] 2022-01-20 18:33:55.118020: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.236084 140191300101952 af.py:224] 2022-01-20 18:33:55.118033: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236149 140191300101952 af.py:224] 2022-01-20 18:33:55.118043: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236207 140191300101952 af.py:224] 2022-01-20 18:33:55.118051: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.236251 140191300101952 af.py:224] 2022-01-20 18:33:55.118063: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236292 140191300101952 af.py:224] 2022-01-20 18:33:55.118072: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.236333 140191300101952 af.py:224] 2022-01-20 18:33:55.118087: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.236374 140191300101952 af.py:224] 2022-01-20 18:33:55.118099: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236414 140191300101952 af.py:224] 2022-01-20 18:33:55.118107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236455 140191300101952 af.py:224] 2022-01-20 18:33:55.118114: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.236496 140191300101952 af.py:224] 2022-01-20 18:33:55.118122: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236536 140191300101952 af.py:224] 2022-01-20 18:33:55.118129: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.236577 140191300101952 af.py:224] 2022-01-20 18:33:55.118143: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.236617 140191300101952 af.py:224] 2022-01-20 18:33:55.118155: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236658 140191300101952 af.py:224] 2022-01-20 18:33:55.118165: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236699 140191300101952 af.py:224] 2022-01-20 18:33:55.118171: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.236739 140191300101952 af.py:224] 2022-01-20 18:33:55.118178: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236779 140191300101952 af.py:224] 2022-01-20 18:33:55.118183: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.236820 140191300101952 af.py:224] 2022-01-20 18:33:55.118191: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.236874 140191300101952 af.py:224] 2022-01-20 18:33:55.118198: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236915 140191300101952 af.py:224] 2022-01-20 18:33:55.118204: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.236959 140191300101952 af.py:224] 2022-01-20 18:33:55.118209: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.237001 140191300101952 af.py:224] 2022-01-20 18:33:55.118218: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237041 140191300101952 af.py:224] 2022-01-20 18:33:55.118228: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.237082 140191300101952 af.py:224] 2022-01-20 18:33:55.118243: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.237122 140191300101952 af.py:224] 2022-01-20 18:33:55.118256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237163 140191300101952 af.py:224] 2022-01-20 18:33:55.118266: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237204 140191300101952 af.py:224] 2022-01-20 18:33:55.118275: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.237244 140191300101952 af.py:224] 2022-01-20 18:33:55.118287: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237284 140191300101952 af.py:224] 2022-01-20 18:33:55.118296: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.237324 140191300101952 af.py:224] 2022-01-20 18:33:55.118309: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.237364 140191300101952 af.py:224] 2022-01-20 18:33:55.118323: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237405 140191300101952 af.py:224] 2022-01-20 18:33:55.118332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237445 140191300101952 af.py:224] 2022-01-20 18:33:55.118341: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.237486 140191300101952 af.py:224] 2022-01-20 18:33:55.118353: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237526 140191300101952 af.py:224] 2022-01-20 18:33:55.118363: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.237567 140191300101952 af.py:224] 2022-01-20 18:33:55.118377: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.237608 140191300101952 af.py:224] 2022-01-20 18:33:55.118387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237652 140191300101952 af.py:224] 2022-01-20 18:33:55.118396: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237692 140191300101952 af.py:224] 2022-01-20 18:33:55.118402: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.237732 140191300101952 af.py:224] 2022-01-20 18:33:55.118409: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237773 140191300101952 af.py:224] 2022-01-20 18:33:55.118414: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.237813 140191300101952 af.py:224] 2022-01-20 18:33:55.118422: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.237854 140191300101952 af.py:224] 2022-01-20 18:33:55.118429: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237894 140191300101952 af.py:224] 2022-01-20 18:33:55.118435: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.237935 140191300101952 af.py:224] 2022-01-20 18:33:55.118440: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.237976 140191300101952 af.py:224] 2022-01-20 18:33:55.118447: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238016 140191300101952 af.py:224] 2022-01-20 18:33:55.118455: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.238056 140191300101952 af.py:224] 2022-01-20 18:33:55.118464: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.238098 140191300101952 af.py:224] 2022-01-20 18:33:55.118471: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238139 140191300101952 af.py:224] 2022-01-20 18:33:55.118478: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238179 140191300101952 af.py:224] 2022-01-20 18:33:55.118483: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.238219 140191300101952 af.py:224] 2022-01-20 18:33:55.118490: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238260 140191300101952 af.py:224] 2022-01-20 18:33:55.118495: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.238301 140191300101952 af.py:224] 2022-01-20 18:33:55.118503: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.238345 140191300101952 af.py:224] 2022-01-20 18:33:55.118510: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238386 140191300101952 af.py:224] 2022-01-20 18:33:55.118516: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238427 140191300101952 af.py:224] 2022-01-20 18:33:55.118521: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.238480 140191300101952 af.py:224] 2022-01-20 18:33:55.118528: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238522 140191300101952 af.py:224] 2022-01-20 18:33:55.118533: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.238562 140191300101952 af.py:224] 2022-01-20 18:33:55.118541: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.238603 140191300101952 af.py:224] 2022-01-20 18:33:55.118548: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238644 140191300101952 af.py:224] 2022-01-20 18:33:55.118555: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238684 140191300101952 af.py:224] 2022-01-20 18:33:55.118561: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.238725 140191300101952 af.py:224] 2022-01-20 18:33:55.118571: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238765 140191300101952 af.py:224] 2022-01-20 18:33:55.118579: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.238806 140191300101952 af.py:224] 2022-01-20 18:33:55.118590: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.238847 140191300101952 af.py:224] 2022-01-20 18:33:55.118598: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238888 140191300101952 af.py:224] 2022-01-20 18:33:55.118604: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.238928 140191300101952 af.py:224] 2022-01-20 18:33:55.118609: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.238969 140191300101952 af.py:224] 2022-01-20 18:33:55.118615: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239015 140191300101952 af.py:224] 2022-01-20 18:33:55.118621: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.239056 140191300101952 af.py:224] 2022-01-20 18:33:55.118629: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.239097 140191300101952 af.py:224] 2022-01-20 18:33:55.118636: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239138 140191300101952 af.py:224] 2022-01-20 18:33:55.118646: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239178 140191300101952 af.py:224] 2022-01-20 18:33:55.118655: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.239219 140191300101952 af.py:224] 2022-01-20 18:33:55.118667: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239259 140191300101952 af.py:224] 2022-01-20 18:33:55.118676: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.239300 140191300101952 af.py:224] 2022-01-20 18:33:55.118690: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.239340 140191300101952 af.py:224] 2022-01-20 18:33:55.118704: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239380 140191300101952 af.py:224] 2022-01-20 18:33:55.118715: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239421 140191300101952 af.py:224] 2022-01-20 18:33:55.118725: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.239461 140191300101952 af.py:224] 2022-01-20 18:33:55.118738: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239502 140191300101952 af.py:224] 2022-01-20 18:33:55.118748: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.239542 140191300101952 af.py:224] 2022-01-20 18:33:55.118762: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.239583 140191300101952 af.py:224] 2022-01-20 18:33:55.118774: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239623 140191300101952 af.py:224] 2022-01-20 18:33:55.118786: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239663 140191300101952 af.py:224] 2022-01-20 18:33:55.118796: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.239706 140191300101952 af.py:224] 2022-01-20 18:33:55.118809: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239748 140191300101952 af.py:224] 2022-01-20 18:33:55.118819: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.239788 140191300101952 af.py:224] 2022-01-20 18:33:55.118834: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.239829 140191300101952 af.py:224] 2022-01-20 18:33:55.118849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239870 140191300101952 af.py:224] 2022-01-20 18:33:55.118860: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239910 140191300101952 af.py:224] 2022-01-20 18:33:55.118870: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.239951 140191300101952 af.py:224] 2022-01-20 18:33:55.118882: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.239991 140191300101952 af.py:224] 2022-01-20 18:33:55.118891: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.240032 140191300101952 af.py:224] 2022-01-20 18:33:55.118906: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.240087 140191300101952 af.py:224] 2022-01-20 18:33:55.118948: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240131 140191300101952 af.py:224] 2022-01-20 18:33:55.118959: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240173 140191300101952 af.py:224] 2022-01-20 18:33:55.118964: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.240214 140191300101952 af.py:224] 2022-01-20 18:33:55.118974: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240255 140191300101952 af.py:224] 2022-01-20 18:33:55.118983: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.240296 140191300101952 af.py:224] 2022-01-20 18:33:55.118998: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.240336 140191300101952 af.py:224] 2022-01-20 18:33:55.119010: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240377 140191300101952 af.py:224] 2022-01-20 18:33:55.119021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240421 140191300101952 af.py:224] 2022-01-20 18:33:55.119030: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.240462 140191300101952 af.py:224] 2022-01-20 18:33:55.119042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240502 140191300101952 af.py:224] 2022-01-20 18:33:55.119051: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.240542 140191300101952 af.py:224] 2022-01-20 18:33:55.119065: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.240583 140191300101952 af.py:224] 2022-01-20 18:33:55.119078: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240624 140191300101952 af.py:224] 2022-01-20 18:33:55.119089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240665 140191300101952 af.py:224] 2022-01-20 18:33:55.119095: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.240705 140191300101952 af.py:224] 2022-01-20 18:33:55.119102: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240746 140191300101952 af.py:224] 2022-01-20 18:33:55.119107: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.240787 140191300101952 af.py:224] 2022-01-20 18:33:55.119116: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.240827 140191300101952 af.py:224] 2022-01-20 18:33:55.119126: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240868 140191300101952 af.py:224] 2022-01-20 18:33:55.119135: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240908 140191300101952 af.py:224] 2022-01-20 18:33:55.119144: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.240949 140191300101952 af.py:224] 2022-01-20 18:33:55.119156: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.240989 140191300101952 af.py:224] 2022-01-20 18:33:55.119164: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.241030 140191300101952 af.py:224] 2022-01-20 18:33:55.119180: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.241070 140191300101952 af.py:224] 2022-01-20 18:33:55.119192: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241114 140191300101952 af.py:224] 2022-01-20 18:33:55.119203: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241155 140191300101952 af.py:224] 2022-01-20 18:33:55.119212: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.241196 140191300101952 af.py:224] 2022-01-20 18:33:55.119225: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241238 140191300101952 af.py:224] 2022-01-20 18:33:55.119235: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.241279 140191300101952 af.py:224] 2022-01-20 18:33:55.119249: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.241320 140191300101952 af.py:224] 2022-01-20 18:33:55.119261: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241361 140191300101952 af.py:224] 2022-01-20 18:33:55.119271: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241402 140191300101952 af.py:224] 2022-01-20 18:33:55.119281: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.241442 140191300101952 af.py:224] 2022-01-20 18:33:55.119294: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241483 140191300101952 af.py:224] 2022-01-20 18:33:55.119303: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.241524 140191300101952 af.py:224] 2022-01-20 18:33:55.119316: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.241566 140191300101952 af.py:224] 2022-01-20 18:33:55.119325: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241607 140191300101952 af.py:224] 2022-01-20 18:33:55.119331: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241648 140191300101952 af.py:224] 2022-01-20 18:33:55.119337: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.241689 140191300101952 af.py:224] 2022-01-20 18:33:55.119343: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241748 140191300101952 af.py:224] 2022-01-20 18:33:55.119349: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.241794 140191300101952 af.py:224] 2022-01-20 18:33:55.119357: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.241835 140191300101952 af.py:224] 2022-01-20 18:33:55.119366: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241876 140191300101952 af.py:224] 2022-01-20 18:33:55.119375: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.241951 140191300101952 af.py:224] 2022-01-20 18:33:55.119384: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.242010 140191300101952 af.py:224] 2022-01-20 18:33:55.119396: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242066 140191300101952 af.py:224] 2022-01-20 18:33:55.119406: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.242106 140191300101952 af.py:224] 2022-01-20 18:33:55.119421: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.242147 140191300101952 af.py:224] 2022-01-20 18:33:55.119434: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242189 140191300101952 af.py:224] 2022-01-20 18:33:55.119445: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242229 140191300101952 af.py:224] 2022-01-20 18:33:55.119454: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.242270 140191300101952 af.py:224] 2022-01-20 18:33:55.119466: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242310 140191300101952 af.py:224] 2022-01-20 18:33:55.119474: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.242351 140191300101952 af.py:224] 2022-01-20 18:33:55.119487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.242391 140191300101952 af.py:224] 2022-01-20 18:33:55.119501: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242431 140191300101952 af.py:224] 2022-01-20 18:33:55.119512: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242472 140191300101952 af.py:224] 2022-01-20 18:33:55.119518: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.242512 140191300101952 af.py:224] 2022-01-20 18:33:55.119525: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242556 140191300101952 af.py:224] 2022-01-20 18:33:55.119531: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.242597 140191300101952 af.py:224] 2022-01-20 18:33:55.119539: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.242638 140191300101952 af.py:224] 2022-01-20 18:33:55.119547: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242678 140191300101952 af.py:224] 2022-01-20 18:33:55.119553: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242719 140191300101952 af.py:224] 2022-01-20 18:33:55.119558: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.242760 140191300101952 af.py:224] 2022-01-20 18:33:55.119568: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242802 140191300101952 af.py:224] 2022-01-20 18:33:55.119578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.242843 140191300101952 af.py:224] 2022-01-20 18:33:55.119592: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.242884 140191300101952 af.py:224] 2022-01-20 18:33:55.119604: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242925 140191300101952 af.py:224] 2022-01-20 18:33:55.119615: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.242966 140191300101952 af.py:224] 2022-01-20 18:33:55.119625: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.243007 140191300101952 af.py:224] 2022-01-20 18:33:55.119637: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243047 140191300101952 af.py:224] 2022-01-20 18:33:55.119647: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.243088 140191300101952 af.py:224] 2022-01-20 18:33:55.119662: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.243128 140191300101952 af.py:224] 2022-01-20 18:33:55.119675: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243169 140191300101952 af.py:224] 2022-01-20 18:33:55.119686: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243209 140191300101952 af.py:224] 2022-01-20 18:33:55.119696: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.243252 140191300101952 af.py:224] 2022-01-20 18:33:55.119708: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243294 140191300101952 af.py:224] 2022-01-20 18:33:55.119718: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.243334 140191300101952 af.py:224] 2022-01-20 18:33:55.119733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.243375 140191300101952 af.py:224] 2022-01-20 18:33:55.119745: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243427 140191300101952 af.py:224] 2022-01-20 18:33:55.119753: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243469 140191300101952 af.py:224] 2022-01-20 18:33:55.119758: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.243510 140191300101952 af.py:224] 2022-01-20 18:33:55.119765: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243552 140191300101952 af.py:224] 2022-01-20 18:33:55.119770: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.243592 140191300101952 af.py:224] 2022-01-20 18:33:55.119779: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.243633 140191300101952 af.py:224] 2022-01-20 18:33:55.119785: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243674 140191300101952 af.py:224] 2022-01-20 18:33:55.119791: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243715 140191300101952 af.py:224] 2022-01-20 18:33:55.119797: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.243755 140191300101952 af.py:224] 2022-01-20 18:33:55.119803: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243796 140191300101952 af.py:224] 2022-01-20 18:33:55.119809: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.243836 140191300101952 af.py:224] 2022-01-20 18:33:55.119817: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.243876 140191300101952 af.py:224] 2022-01-20 18:33:55.119823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243917 140191300101952 af.py:224] 2022-01-20 18:33:55.119829: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.243961 140191300101952 af.py:224] 2022-01-20 18:33:55.119834: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.244002 140191300101952 af.py:224] 2022-01-20 18:33:55.119841: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244043 140191300101952 af.py:224] 2022-01-20 18:33:55.119846: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.244083 140191300101952 af.py:224] 2022-01-20 18:33:55.119855: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.244127 140191300101952 af.py:224] 2022-01-20 18:33:55.119862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244169 140191300101952 af.py:224] 2022-01-20 18:33:55.119868: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244210 140191300101952 af.py:224] 2022-01-20 18:33:55.119874: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.244251 140191300101952 af.py:224] 2022-01-20 18:33:55.119885: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244292 140191300101952 af.py:224] 2022-01-20 18:33:55.119894: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.244332 140191300101952 af.py:224] 2022-01-20 18:33:55.119905: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.244372 140191300101952 af.py:224] 2022-01-20 18:33:55.119913: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244413 140191300101952 af.py:224] 2022-01-20 18:33:55.119919: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244454 140191300101952 af.py:224] 2022-01-20 18:33:55.119925: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.244494 140191300101952 af.py:224] 2022-01-20 18:33:55.119931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244534 140191300101952 af.py:224] 2022-01-20 18:33:55.119936: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.244575 140191300101952 af.py:224] 2022-01-20 18:33:55.119945: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.244615 140191300101952 af.py:224] 2022-01-20 18:33:55.119952: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244659 140191300101952 af.py:224] 2022-01-20 18:33:55.119959: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244700 140191300101952 af.py:224] 2022-01-20 18:33:55.119964: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.244741 140191300101952 af.py:224] 2022-01-20 18:33:55.119974: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244781 140191300101952 af.py:224] 2022-01-20 18:33:55.119983: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.244823 140191300101952 af.py:224] 2022-01-20 18:33:55.119997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.244863 140191300101952 af.py:224] 2022-01-20 18:33:55.120010: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244904 140191300101952 af.py:224] 2022-01-20 18:33:55.120020: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.244944 140191300101952 af.py:224] 2022-01-20 18:33:55.120029: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.244985 140191300101952 af.py:224] 2022-01-20 18:33:55.120041: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245038 140191300101952 af.py:224] 2022-01-20 18:33:55.120050: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.245080 140191300101952 af.py:224] 2022-01-20 18:33:55.120066: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.245120 140191300101952 af.py:224] 2022-01-20 18:33:55.120079: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245161 140191300101952 af.py:224] 2022-01-20 18:33:55.120090: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245201 140191300101952 af.py:224] 2022-01-20 18:33:55.120099: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.245242 140191300101952 af.py:224] 2022-01-20 18:33:55.120109: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245283 140191300101952 af.py:224] 2022-01-20 18:33:55.120122: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.245327 140191300101952 af.py:224] 2022-01-20 18:33:55.120138: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.245368 140191300101952 af.py:224] 2022-01-20 18:33:55.120151: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245409 140191300101952 af.py:224] 2022-01-20 18:33:55.120162: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245450 140191300101952 af.py:224] 2022-01-20 18:33:55.120170: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.245491 140191300101952 af.py:224] 2022-01-20 18:33:55.120182: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245531 140191300101952 af.py:224] 2022-01-20 18:33:55.120192: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.245572 140191300101952 af.py:224] 2022-01-20 18:33:55.120204: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.245613 140191300101952 af.py:224] 2022-01-20 18:33:55.120212: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245653 140191300101952 af.py:224] 2022-01-20 18:33:55.120219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245693 140191300101952 af.py:224] 2022-01-20 18:33:55.120225: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.245733 140191300101952 af.py:224] 2022-01-20 18:33:55.120231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245774 140191300101952 af.py:224] 2022-01-20 18:33:55.120236: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.245815 140191300101952 af.py:224] 2022-01-20 18:33:55.120245: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.245855 140191300101952 af.py:224] 2022-01-20 18:33:55.120256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245895 140191300101952 af.py:224] 2022-01-20 18:33:55.120267: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.245936 140191300101952 af.py:224] 2022-01-20 18:33:55.120276: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.245977 140191300101952 af.py:224] 2022-01-20 18:33:55.120289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246021 140191300101952 af.py:224] 2022-01-20 18:33:55.120299: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.246062 140191300101952 af.py:224] 2022-01-20 18:33:55.120314: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.246102 140191300101952 af.py:224] 2022-01-20 18:33:55.120327: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246142 140191300101952 af.py:224] 2022-01-20 18:33:55.120337: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246182 140191300101952 af.py:224] 2022-01-20 18:33:55.120345: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.246223 140191300101952 af.py:224] 2022-01-20 18:33:55.120358: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246263 140191300101952 af.py:224] 2022-01-20 18:33:55.120366: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.246304 140191300101952 af.py:224] 2022-01-20 18:33:55.120379: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.246345 140191300101952 af.py:224] 2022-01-20 18:33:55.120392: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246384 140191300101952 af.py:224] 2022-01-20 18:33:55.120402: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246425 140191300101952 af.py:224] 2022-01-20 18:33:55.120430: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.246465 140191300101952 af.py:224] 2022-01-20 18:33:55.120443: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246506 140191300101952 af.py:224] 2022-01-20 18:33:55.120451: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.246547 140191300101952 af.py:224] 2022-01-20 18:33:55.120465: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.246587 140191300101952 af.py:224] 2022-01-20 18:33:55.120477: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246641 140191300101952 af.py:224] 2022-01-20 18:33:55.120488: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246683 140191300101952 af.py:224] 2022-01-20 18:33:55.120496: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.246728 140191300101952 af.py:224] 2022-01-20 18:33:55.120508: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246769 140191300101952 af.py:224] 2022-01-20 18:33:55.120517: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.246810 140191300101952 af.py:224] 2022-01-20 18:33:55.120528: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.246850 140191300101952 af.py:224] 2022-01-20 18:33:55.120536: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246891 140191300101952 af.py:224] 2022-01-20 18:33:55.120542: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.246932 140191300101952 af.py:224] 2022-01-20 18:33:55.120547: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.246973 140191300101952 af.py:224] 2022-01-20 18:33:55.120555: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247014 140191300101952 af.py:224] 2022-01-20 18:33:55.120560: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.247054 140191300101952 af.py:224] 2022-01-20 18:33:55.120568: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.247095 140191300101952 af.py:224] 2022-01-20 18:33:55.120578: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247136 140191300101952 af.py:224] 2022-01-20 18:33:55.120588: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247176 140191300101952 af.py:224] 2022-01-20 18:33:55.120598: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.247216 140191300101952 af.py:224] 2022-01-20 18:33:55.120611: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247256 140191300101952 af.py:224] 2022-01-20 18:33:55.120620: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.247297 140191300101952 af.py:224] 2022-01-20 18:33:55.120630: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.247337 140191300101952 af.py:224] 2022-01-20 18:33:55.120640: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247377 140191300101952 af.py:224] 2022-01-20 18:33:55.120646: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247421 140191300101952 af.py:224] 2022-01-20 18:33:55.120651: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.247461 140191300101952 af.py:224] 2022-01-20 18:33:55.120658: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247502 140191300101952 af.py:224] 2022-01-20 18:33:55.120664: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.247543 140191300101952 af.py:224] 2022-01-20 18:33:55.120674: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.247583 140191300101952 af.py:224] 2022-01-20 18:33:55.120685: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247624 140191300101952 af.py:224] 2022-01-20 18:33:55.120696: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247664 140191300101952 af.py:224] 2022-01-20 18:33:55.120704: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.247704 140191300101952 af.py:224] 2022-01-20 18:33:55.120715: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247745 140191300101952 af.py:224] 2022-01-20 18:33:55.120725: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.247786 140191300101952 af.py:224] 2022-01-20 18:33:55.120740: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.247826 140191300101952 af.py:224] 2022-01-20 18:33:55.120751: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247867 140191300101952 af.py:224] 2022-01-20 18:33:55.120762: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247908 140191300101952 af.py:224] 2022-01-20 18:33:55.120771: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.247948 140191300101952 af.py:224] 2022-01-20 18:33:55.120783: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.247989 140191300101952 af.py:224] 2022-01-20 18:33:55.120792: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.248030 140191300101952 af.py:224] 2022-01-20 18:33:55.120806: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.248070 140191300101952 af.py:224] 2022-01-20 18:33:55.120819: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.248116 140191300101952 af.py:224] 2022-01-20 18:33:55.120830: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.248158 140191300101952 af.py:224] 2022-01-20 18:33:55.120839: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.248199 140191300101952 af.py:224] 2022-01-20 18:33:55.120850: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.248239 140191300101952 af.py:224] 2022-01-20 18:33:55.120859: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.248295 140191300101952 af.py:224] 2022-01-20 18:33:55.120874: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.248338 140191300101952 af.py:224] 2022-01-20 18:33:55.120886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.248420 140191300101952 af.py:224] 2022-01-20 18:33:55.120894: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.248501 140191300101952 af.py:224] 2022-01-20 18:33:55.120899: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.248558 140191300101952 af.py:224] 2022-01-20 18:33:55.120906: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.248631 140191300101952 af.py:224] 2022-01-20 18:33:55.120911: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.248707 140191300101952 af.py:224] 2022-01-20 18:33:55.120919: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.248763 140191300101952 af.py:224] 2022-01-20 18:33:55.120927: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.248803 140191300101952 af.py:224] 2022-01-20 18:33:55.120936: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.248844 140191300101952 af.py:224] 2022-01-20 18:33:55.120945: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.248884 140191300101952 af.py:224] 2022-01-20 18:33:55.120953: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.248925 140191300101952 af.py:224] 2022-01-20 18:33:55.120961: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.248969 140191300101952 af.py:224] 2022-01-20 18:33:55.120970: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.249009 140191300101952 af.py:224] 2022-01-20 18:33:55.120978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249050 140191300101952 af.py:224] 2022-01-20 18:33:55.120985: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249091 140191300101952 af.py:224] 2022-01-20 18:33:55.120994: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.249132 140191300101952 af.py:224] 2022-01-20 18:33:55.121005: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249171 140191300101952 af.py:224] 2022-01-20 18:33:55.121015: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.249212 140191300101952 af.py:224] 2022-01-20 18:33:55.121029: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.249253 140191300101952 af.py:224] 2022-01-20 18:33:55.121041: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249294 140191300101952 af.py:224] 2022-01-20 18:33:55.121049: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249334 140191300101952 af.py:224] 2022-01-20 18:33:55.121058: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.249375 140191300101952 af.py:224] 2022-01-20 18:33:55.121069: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249416 140191300101952 af.py:224] 2022-01-20 18:33:55.121079: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.249457 140191300101952 af.py:224] 2022-01-20 18:33:55.121094: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.249497 140191300101952 af.py:224] 2022-01-20 18:33:55.121106: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249538 140191300101952 af.py:224] 2022-01-20 18:33:55.121115: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249579 140191300101952 af.py:224] 2022-01-20 18:33:55.121124: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.249619 140191300101952 af.py:224] 2022-01-20 18:33:55.121136: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249664 140191300101952 af.py:224] 2022-01-20 18:33:55.121146: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.249705 140191300101952 af.py:224] 2022-01-20 18:33:55.121161: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.249746 140191300101952 af.py:224] 2022-01-20 18:33:55.121175: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249787 140191300101952 af.py:224] 2022-01-20 18:33:55.121186: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249827 140191300101952 af.py:224] 2022-01-20 18:33:55.121195: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.249867 140191300101952 af.py:224] 2022-01-20 18:33:55.121207: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.249908 140191300101952 af.py:224] 2022-01-20 18:33:55.121216: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.249948 140191300101952 af.py:224] 2022-01-20 18:33:55.121228: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.249988 140191300101952 af.py:224] 2022-01-20 18:33:55.121237: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250042 140191300101952 af.py:224] 2022-01-20 18:33:55.121244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250084 140191300101952 af.py:224] 2022-01-20 18:33:55.121250: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.250125 140191300101952 af.py:224] 2022-01-20 18:33:55.121256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250165 140191300101952 af.py:224] 2022-01-20 18:33:55.121261: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.250206 140191300101952 af.py:224] 2022-01-20 18:33:55.121270: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.250246 140191300101952 af.py:224] 2022-01-20 18:33:55.121281: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250287 140191300101952 af.py:224] 2022-01-20 18:33:55.121289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250327 140191300101952 af.py:224] 2022-01-20 18:33:55.121294: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.250372 140191300101952 af.py:224] 2022-01-20 18:33:55.121301: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250412 140191300101952 af.py:224] 2022-01-20 18:33:55.121308: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.250453 140191300101952 af.py:224] 2022-01-20 18:33:55.121321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.250493 140191300101952 af.py:224] 2022-01-20 18:33:55.121332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250534 140191300101952 af.py:224] 2022-01-20 18:33:55.121340: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250574 140191300101952 af.py:224] 2022-01-20 18:33:55.121348: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.250615 140191300101952 af.py:224] 2022-01-20 18:33:55.121358: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250655 140191300101952 af.py:224] 2022-01-20 18:33:55.121365: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.250695 140191300101952 af.py:224] 2022-01-20 18:33:55.121378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.250736 140191300101952 af.py:224] 2022-01-20 18:33:55.121390: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250777 140191300101952 af.py:224] 2022-01-20 18:33:55.121398: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250817 140191300101952 af.py:224] 2022-01-20 18:33:55.121404: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.250858 140191300101952 af.py:224] 2022-01-20 18:33:55.121412: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.250898 140191300101952 af.py:224] 2022-01-20 18:33:55.121417: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.250939 140191300101952 af.py:224] 2022-01-20 18:33:55.121428: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.250979 140191300101952 af.py:224] 2022-01-20 18:33:55.121436: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251020 140191300101952 af.py:224] 2022-01-20 18:33:55.121445: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251064 140191300101952 af.py:224] 2022-01-20 18:33:55.121451: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.251106 140191300101952 af.py:224] 2022-01-20 18:33:55.121459: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251147 140191300101952 af.py:224] 2022-01-20 18:33:55.121465: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.251186 140191300101952 af.py:224] 2022-01-20 18:33:55.121475: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.251227 140191300101952 af.py:224] 2022-01-20 18:33:55.121483: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251267 140191300101952 af.py:224] 2022-01-20 18:33:55.121489: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251308 140191300101952 af.py:224] 2022-01-20 18:33:55.121495: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.251348 140191300101952 af.py:224] 2022-01-20 18:33:55.121503: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251389 140191300101952 af.py:224] 2022-01-20 18:33:55.121508: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.251429 140191300101952 af.py:224] 2022-01-20 18:33:55.121519: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.251470 140191300101952 af.py:224] 2022-01-20 18:33:55.121527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251511 140191300101952 af.py:224] 2022-01-20 18:33:55.121534: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251551 140191300101952 af.py:224] 2022-01-20 18:33:55.121540: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.251592 140191300101952 af.py:224] 2022-01-20 18:33:55.121548: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251633 140191300101952 af.py:224] 2022-01-20 18:33:55.121553: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.251687 140191300101952 af.py:224] 2022-01-20 18:33:55.121563: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.251729 140191300101952 af.py:224] 2022-01-20 18:33:55.121572: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251774 140191300101952 af.py:224] 2022-01-20 18:33:55.121580: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251815 140191300101952 af.py:224] 2022-01-20 18:33:55.121585: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.251855 140191300101952 af.py:224] 2022-01-20 18:33:55.121594: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.251896 140191300101952 af.py:224] 2022-01-20 18:33:55.121600: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.251936 140191300101952 af.py:224] 2022-01-20 18:33:55.121629: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.251976 140191300101952 af.py:224] 2022-01-20 18:33:55.121639: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252017 140191300101952 af.py:224] 2022-01-20 18:33:55.121646: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252058 140191300101952 af.py:224] 2022-01-20 18:33:55.121652: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.252097 140191300101952 af.py:224] 2022-01-20 18:33:55.121660: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252140 140191300101952 af.py:224] 2022-01-20 18:33:55.121666: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.252182 140191300101952 af.py:224] 2022-01-20 18:33:55.121677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.252222 140191300101952 af.py:224] 2022-01-20 18:33:55.121685: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252264 140191300101952 af.py:224] 2022-01-20 18:33:55.121692: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252304 140191300101952 af.py:224] 2022-01-20 18:33:55.121698: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.252344 140191300101952 af.py:224] 2022-01-20 18:33:55.121706: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252385 140191300101952 af.py:224] 2022-01-20 18:33:55.121711: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.252429 140191300101952 af.py:224] 2022-01-20 18:33:55.121722: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.252470 140191300101952 af.py:224] 2022-01-20 18:33:55.121731: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252511 140191300101952 af.py:224] 2022-01-20 18:33:55.121738: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252551 140191300101952 af.py:224] 2022-01-20 18:33:55.121744: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.252591 140191300101952 af.py:224] 2022-01-20 18:33:55.121752: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252632 140191300101952 af.py:224] 2022-01-20 18:33:55.121758: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.252673 140191300101952 af.py:224] 2022-01-20 18:33:55.121769: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.252714 140191300101952 af.py:224] 2022-01-20 18:33:55.121778: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252754 140191300101952 af.py:224] 2022-01-20 18:33:55.121784: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252796 140191300101952 af.py:224] 2022-01-20 18:33:55.121790: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.252836 140191300101952 af.py:224] 2022-01-20 18:33:55.121798: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252877 140191300101952 af.py:224] 2022-01-20 18:33:55.121804: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.252918 140191300101952 af.py:224] 2022-01-20 18:33:55.121815: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.252959 140191300101952 af.py:224] 2022-01-20 18:33:55.121823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.252999 140191300101952 af.py:224] 2022-01-20 18:33:55.121830: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253040 140191300101952 af.py:224] 2022-01-20 18:33:55.121835: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.253080 140191300101952 af.py:224] 2022-01-20 18:33:55.121843: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253123 140191300101952 af.py:224] 2022-01-20 18:33:55.121849: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.253164 140191300101952 af.py:224] 2022-01-20 18:33:55.121876: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.253205 140191300101952 af.py:224] 2022-01-20 18:33:55.121885: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253245 140191300101952 af.py:224] 2022-01-20 18:33:55.121892: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253300 140191300101952 af.py:224] 2022-01-20 18:33:55.121898: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.253342 140191300101952 af.py:224] 2022-01-20 18:33:55.121906: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253383 140191300101952 af.py:224] 2022-01-20 18:33:55.121911: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.253424 140191300101952 af.py:224] 2022-01-20 18:33:55.121922: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.253465 140191300101952 af.py:224] 2022-01-20 18:33:55.121931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253505 140191300101952 af.py:224] 2022-01-20 18:33:55.121937: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253546 140191300101952 af.py:224] 2022-01-20 18:33:55.121943: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.253588 140191300101952 af.py:224] 2022-01-20 18:33:55.121951: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253629 140191300101952 af.py:224] 2022-01-20 18:33:55.121957: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.253669 140191300101952 af.py:224] 2022-01-20 18:33:55.121968: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.253710 140191300101952 af.py:224] 2022-01-20 18:33:55.121976: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253751 140191300101952 af.py:224] 2022-01-20 18:33:55.121983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253792 140191300101952 af.py:224] 2022-01-20 18:33:55.121988: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.253836 140191300101952 af.py:224] 2022-01-20 18:33:55.121997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253877 140191300101952 af.py:224] 2022-01-20 18:33:55.122002: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.253917 140191300101952 af.py:224] 2022-01-20 18:33:55.122013: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.253958 140191300101952 af.py:224] 2022-01-20 18:33:55.122021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.253998 140191300101952 af.py:224] 2022-01-20 18:33:55.122028: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254038 140191300101952 af.py:224] 2022-01-20 18:33:55.122033: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.254079 140191300101952 af.py:224] 2022-01-20 18:33:55.122041: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254121 140191300101952 af.py:224] 2022-01-20 18:33:55.122047: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.254161 140191300101952 af.py:224] 2022-01-20 18:33:55.122057: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.254201 140191300101952 af.py:224] 2022-01-20 18:33:55.122065: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254242 140191300101952 af.py:224] 2022-01-20 18:33:55.122071: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254282 140191300101952 af.py:224] 2022-01-20 18:33:55.122077: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.254323 140191300101952 af.py:224] 2022-01-20 18:33:55.122085: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254364 140191300101952 af.py:224] 2022-01-20 18:33:55.122091: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.254405 140191300101952 af.py:224] 2022-01-20 18:33:55.122119: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.254446 140191300101952 af.py:224] 2022-01-20 18:33:55.122128: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254487 140191300101952 af.py:224] 2022-01-20 18:33:55.122135: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254532 140191300101952 af.py:224] 2022-01-20 18:33:55.122141: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.254573 140191300101952 af.py:224] 2022-01-20 18:33:55.122149: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254613 140191300101952 af.py:224] 2022-01-20 18:33:55.122155: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.254653 140191300101952 af.py:224] 2022-01-20 18:33:55.122166: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.254694 140191300101952 af.py:224] 2022-01-20 18:33:55.122175: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254734 140191300101952 af.py:224] 2022-01-20 18:33:55.122183: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254775 140191300101952 af.py:224] 2022-01-20 18:33:55.122189: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.254816 140191300101952 af.py:224] 2022-01-20 18:33:55.122198: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.254856 140191300101952 af.py:224] 2022-01-20 18:33:55.122204: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.254913 140191300101952 af.py:224] 2022-01-20 18:33:55.122212: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.254977 140191300101952 af.py:224] 2022-01-20 18:33:55.122220: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255018 140191300101952 af.py:224] 2022-01-20 18:33:55.122227: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255091 140191300101952 af.py:224] 2022-01-20 18:33:55.122231: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.255148 140191300101952 af.py:224] 2022-01-20 18:33:55.122238: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255189 140191300101952 af.py:224] 2022-01-20 18:33:55.122242: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.255247 140191300101952 af.py:224] 2022-01-20 18:33:55.122250: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.255307 140191300101952 af.py:224] 2022-01-20 18:33:55.122256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255349 140191300101952 af.py:224] 2022-01-20 18:33:55.122262: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255390 140191300101952 af.py:224] 2022-01-20 18:33:55.122267: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.255431 140191300101952 af.py:224] 2022-01-20 18:33:55.122273: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255472 140191300101952 af.py:224] 2022-01-20 18:33:55.122278: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.255512 140191300101952 af.py:224] 2022-01-20 18:33:55.122285: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.255553 140191300101952 af.py:224] 2022-01-20 18:33:55.122292: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255594 140191300101952 af.py:224] 2022-01-20 18:33:55.122298: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255635 140191300101952 af.py:224] 2022-01-20 18:33:55.122303: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.255676 140191300101952 af.py:224] 2022-01-20 18:33:55.122310: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255717 140191300101952 af.py:224] 2022-01-20 18:33:55.122315: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.255758 140191300101952 af.py:224] 2022-01-20 18:33:55.122323: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.255798 140191300101952 af.py:224] 2022-01-20 18:33:55.122330: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255838 140191300101952 af.py:224] 2022-01-20 18:33:55.122336: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255879 140191300101952 af.py:224] 2022-01-20 18:33:55.122341: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.255919 140191300101952 af.py:224] 2022-01-20 18:33:55.122347: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.255959 140191300101952 af.py:224] 2022-01-20 18:33:55.122352: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.256003 140191300101952 af.py:224] 2022-01-20 18:33:55.122361: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.256043 140191300101952 af.py:224] 2022-01-20 18:33:55.122368: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256084 140191300101952 af.py:224] 2022-01-20 18:33:55.122374: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256164 140191300101952 af.py:224] 2022-01-20 18:33:55.122379: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.256240 140191300101952 af.py:224] 2022-01-20 18:33:55.122386: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256281 140191300101952 af.py:224] 2022-01-20 18:33:55.122391: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.256322 140191300101952 af.py:224] 2022-01-20 18:33:55.122398: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.256363 140191300101952 af.py:224] 2022-01-20 18:33:55.122405: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256404 140191300101952 af.py:224] 2022-01-20 18:33:55.122410: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256444 140191300101952 af.py:224] 2022-01-20 18:33:55.122415: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.256485 140191300101952 af.py:224] 2022-01-20 18:33:55.122421: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256526 140191300101952 af.py:224] 2022-01-20 18:33:55.122426: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.256567 140191300101952 af.py:224] 2022-01-20 18:33:55.122435: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.256607 140191300101952 af.py:224] 2022-01-20 18:33:55.122442: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256647 140191300101952 af.py:224] 2022-01-20 18:33:55.122448: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256688 140191300101952 af.py:224] 2022-01-20 18:33:55.122468: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.256743 140191300101952 af.py:224] 2022-01-20 18:33:55.122476: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256788 140191300101952 af.py:224] 2022-01-20 18:33:55.122482: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.256829 140191300101952 af.py:224] 2022-01-20 18:33:55.122490: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.256870 140191300101952 af.py:224] 2022-01-20 18:33:55.122497: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256911 140191300101952 af.py:224] 2022-01-20 18:33:55.122503: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.256952 140191300101952 af.py:224] 2022-01-20 18:33:55.122508: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.256992 140191300101952 af.py:224] 2022-01-20 18:33:55.122514: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257032 140191300101952 af.py:224] 2022-01-20 18:33:55.122519: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.257072 140191300101952 af.py:224] 2022-01-20 18:33:55.122528: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.257112 140191300101952 af.py:224] 2022-01-20 18:33:55.122536: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257153 140191300101952 af.py:224] 2022-01-20 18:33:55.122542: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257194 140191300101952 af.py:224] 2022-01-20 18:33:55.122547: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.257235 140191300101952 af.py:224] 2022-01-20 18:33:55.122554: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257276 140191300101952 af.py:224] 2022-01-20 18:33:55.122559: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.257317 140191300101952 af.py:224] 2022-01-20 18:33:55.122567: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.257357 140191300101952 af.py:224] 2022-01-20 18:33:55.122574: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257397 140191300101952 af.py:224] 2022-01-20 18:33:55.122580: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257438 140191300101952 af.py:224] 2022-01-20 18:33:55.122585: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.257483 140191300101952 af.py:224] 2022-01-20 18:33:55.122592: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257523 140191300101952 af.py:224] 2022-01-20 18:33:55.122597: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.257563 140191300101952 af.py:224] 2022-01-20 18:33:55.122605: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.257603 140191300101952 af.py:224] 2022-01-20 18:33:55.122612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257643 140191300101952 af.py:224] 2022-01-20 18:33:55.122619: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257684 140191300101952 af.py:224] 2022-01-20 18:33:55.122624: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.257725 140191300101952 af.py:224] 2022-01-20 18:33:55.122631: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257765 140191300101952 af.py:224] 2022-01-20 18:33:55.122636: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.257805 140191300101952 af.py:224] 2022-01-20 18:33:55.122644: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.257846 140191300101952 af.py:224] 2022-01-20 18:33:55.122651: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257886 140191300101952 af.py:224] 2022-01-20 18:33:55.122658: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.257927 140191300101952 af.py:224] 2022-01-20 18:33:55.122663: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.257967 140191300101952 af.py:224] 2022-01-20 18:33:55.122670: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258008 140191300101952 af.py:224] 2022-01-20 18:33:55.122675: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.258049 140191300101952 af.py:224] 2022-01-20 18:33:55.122683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.258090 140191300101952 af.py:224] 2022-01-20 18:33:55.122690: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258134 140191300101952 af.py:224] 2022-01-20 18:33:55.122696: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258175 140191300101952 af.py:224] 2022-01-20 18:33:55.122717: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.258215 140191300101952 af.py:224] 2022-01-20 18:33:55.122724: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258255 140191300101952 af.py:224] 2022-01-20 18:33:55.122729: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.258296 140191300101952 af.py:224] 2022-01-20 18:33:55.122738: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.258349 140191300101952 af.py:224] 2022-01-20 18:33:55.122745: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258391 140191300101952 af.py:224] 2022-01-20 18:33:55.122766: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258433 140191300101952 af.py:224] 2022-01-20 18:33:55.122771: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.258473 140191300101952 af.py:224] 2022-01-20 18:33:55.122778: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258514 140191300101952 af.py:224] 2022-01-20 18:33:55.122783: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.258554 140191300101952 af.py:224] 2022-01-20 18:33:55.122792: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.258594 140191300101952 af.py:224] 2022-01-20 18:33:55.122798: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258636 140191300101952 af.py:224] 2022-01-20 18:33:55.122805: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258677 140191300101952 af.py:224] 2022-01-20 18:33:55.122810: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.258716 140191300101952 af.py:224] 2022-01-20 18:33:55.122817: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258757 140191300101952 af.py:224] 2022-01-20 18:33:55.122822: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.258797 140191300101952 af.py:224] 2022-01-20 18:33:55.122831: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.258841 140191300101952 af.py:224] 2022-01-20 18:33:55.122838: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258882 140191300101952 af.py:224] 2022-01-20 18:33:55.122844: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.258923 140191300101952 af.py:224] 2022-01-20 18:33:55.122849: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.258963 140191300101952 af.py:224] 2022-01-20 18:33:55.122855: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259003 140191300101952 af.py:224] 2022-01-20 18:33:55.122860: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.259044 140191300101952 af.py:224] 2022-01-20 18:33:55.122869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.259085 140191300101952 af.py:224] 2022-01-20 18:33:55.122876: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259125 140191300101952 af.py:224] 2022-01-20 18:33:55.122882: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259166 140191300101952 af.py:224] 2022-01-20 18:33:55.122887: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.259207 140191300101952 af.py:224] 2022-01-20 18:33:55.122893: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259247 140191300101952 af.py:224] 2022-01-20 18:33:55.122899: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.259288 140191300101952 af.py:224] 2022-01-20 18:33:55.122907: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.259329 140191300101952 af.py:224] 2022-01-20 18:33:55.122928: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259370 140191300101952 af.py:224] 2022-01-20 18:33:55.122935: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259411 140191300101952 af.py:224] 2022-01-20 18:33:55.122940: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.259451 140191300101952 af.py:224] 2022-01-20 18:33:55.122947: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259492 140191300101952 af.py:224] 2022-01-20 18:33:55.122952: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.259536 140191300101952 af.py:224] 2022-01-20 18:33:55.122960: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.259576 140191300101952 af.py:224] 2022-01-20 18:33:55.122967: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259617 140191300101952 af.py:224] 2022-01-20 18:33:55.122974: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259658 140191300101952 af.py:224] 2022-01-20 18:33:55.122978: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.259698 140191300101952 af.py:224] 2022-01-20 18:33:55.123000: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259739 140191300101952 af.py:224] 2022-01-20 18:33:55.123005: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.259780 140191300101952 af.py:224] 2022-01-20 18:33:55.123013: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.259820 140191300101952 af.py:224] 2022-01-20 18:33:55.123020: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259861 140191300101952 af.py:224] 2022-01-20 18:33:55.123026: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259902 140191300101952 af.py:224] 2022-01-20 18:33:55.123031: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.259956 140191300101952 af.py:224] 2022-01-20 18:33:55.123037: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.259997 140191300101952 af.py:224] 2022-01-20 18:33:55.123042: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.260037 140191300101952 af.py:224] 2022-01-20 18:33:55.123050: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.260077 140191300101952 af.py:224] 2022-01-20 18:33:55.123058: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260138 140191300101952 af.py:224] 2022-01-20 18:33:55.123065: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260196 140191300101952 af.py:224] 2022-01-20 18:33:55.123070: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.260236 140191300101952 af.py:224] 2022-01-20 18:33:55.123077: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260282 140191300101952 af.py:224] 2022-01-20 18:33:55.123082: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.260323 140191300101952 af.py:224] 2022-01-20 18:33:55.123089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.260364 140191300101952 af.py:224] 2022-01-20 18:33:55.123096: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260404 140191300101952 af.py:224] 2022-01-20 18:33:55.123103: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260445 140191300101952 af.py:224] 2022-01-20 18:33:55.123108: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.260485 140191300101952 af.py:224] 2022-01-20 18:33:55.123129: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260526 140191300101952 af.py:224] 2022-01-20 18:33:55.123135: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.260567 140191300101952 af.py:224] 2022-01-20 18:33:55.123143: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.260607 140191300101952 af.py:224] 2022-01-20 18:33:55.123150: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260648 140191300101952 af.py:224] 2022-01-20 18:33:55.123157: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260688 140191300101952 af.py:224] 2022-01-20 18:33:55.123162: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.260729 140191300101952 af.py:224] 2022-01-20 18:33:55.123169: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260770 140191300101952 af.py:224] 2022-01-20 18:33:55.123174: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.260810 140191300101952 af.py:224] 2022-01-20 18:33:55.123181: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.260850 140191300101952 af.py:224] 2022-01-20 18:33:55.123188: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260891 140191300101952 af.py:224] 2022-01-20 18:33:55.123194: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.260935 140191300101952 af.py:224] 2022-01-20 18:33:55.123199: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.260976 140191300101952 af.py:224] 2022-01-20 18:33:55.123205: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261016 140191300101952 af.py:224] 2022-01-20 18:33:55.123210: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.261057 140191300101952 af.py:224] 2022-01-20 18:33:55.123234: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.261097 140191300101952 af.py:224] 2022-01-20 18:33:55.123241: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261138 140191300101952 af.py:224] 2022-01-20 18:33:55.123248: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261178 140191300101952 af.py:224] 2022-01-20 18:33:55.123253: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.261219 140191300101952 af.py:224] 2022-01-20 18:33:55.123259: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261260 140191300101952 af.py:224] 2022-01-20 18:33:55.123264: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.261301 140191300101952 af.py:224] 2022-01-20 18:33:55.123272: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.261341 140191300101952 af.py:224] 2022-01-20 18:33:55.123279: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261381 140191300101952 af.py:224] 2022-01-20 18:33:55.123285: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261422 140191300101952 af.py:224] 2022-01-20 18:33:55.123290: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.261462 140191300101952 af.py:224] 2022-01-20 18:33:55.123296: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261503 140191300101952 af.py:224] 2022-01-20 18:33:55.123301: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.261543 140191300101952 af.py:224] 2022-01-20 18:33:55.123310: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.261602 140191300101952 af.py:224] 2022-01-20 18:33:55.123316: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261665 140191300101952 af.py:224] 2022-01-20 18:33:55.123323: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261707 140191300101952 af.py:224] 2022-01-20 18:33:55.123341: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.261781 140191300101952 af.py:224] 2022-01-20 18:33:55.123348: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261855 140191300101952 af.py:224] 2022-01-20 18:33:55.123353: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.261896 140191300101952 af.py:224] 2022-01-20 18:33:55.123362: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.261954 140191300101952 af.py:224] 2022-01-20 18:33:55.123369: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.261995 140191300101952 af.py:224] 2022-01-20 18:33:55.123375: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262035 140191300101952 af.py:224] 2022-01-20 18:33:55.123380: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.262075 140191300101952 af.py:224] 2022-01-20 18:33:55.123386: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262116 140191300101952 af.py:224] 2022-01-20 18:33:55.123391: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.262156 140191300101952 af.py:224] 2022-01-20 18:33:55.123399: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.262197 140191300101952 af.py:224] 2022-01-20 18:33:55.123406: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262238 140191300101952 af.py:224] 2022-01-20 18:33:55.123413: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262279 140191300101952 af.py:224] 2022-01-20 18:33:55.123418: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.262319 140191300101952 af.py:224] 2022-01-20 18:33:55.123424: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262360 140191300101952 af.py:224] 2022-01-20 18:33:55.123429: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.262401 140191300101952 af.py:224] 2022-01-20 18:33:55.123437: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.262444 140191300101952 af.py:224] 2022-01-20 18:33:55.123458: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262485 140191300101952 af.py:224] 2022-01-20 18:33:55.123465: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262526 140191300101952 af.py:224] 2022-01-20 18:33:55.123470: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.262567 140191300101952 af.py:224] 2022-01-20 18:33:55.123477: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262607 140191300101952 af.py:224] 2022-01-20 18:33:55.123482: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.262647 140191300101952 af.py:224] 2022-01-20 18:33:55.123490: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.262688 140191300101952 af.py:224] 2022-01-20 18:33:55.123497: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262729 140191300101952 af.py:224] 2022-01-20 18:33:55.123504: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262769 140191300101952 af.py:224] 2022-01-20 18:33:55.123509: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.262809 140191300101952 af.py:224] 2022-01-20 18:33:55.123516: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262850 140191300101952 af.py:224] 2022-01-20 18:33:55.123521: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.262890 140191300101952 af.py:224] 2022-01-20 18:33:55.123529: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.262931 140191300101952 af.py:224] 2022-01-20 18:33:55.123536: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.262971 140191300101952 af.py:224] 2022-01-20 18:33:55.123556: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263011 140191300101952 af.py:224] 2022-01-20 18:33:55.123561: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.263052 140191300101952 af.py:224] 2022-01-20 18:33:55.123568: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263092 140191300101952 af.py:224] 2022-01-20 18:33:55.123573: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.263136 140191300101952 af.py:224] 2022-01-20 18:33:55.123582: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.263176 140191300101952 af.py:224] 2022-01-20 18:33:55.123589: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263217 140191300101952 af.py:224] 2022-01-20 18:33:55.123595: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263257 140191300101952 af.py:224] 2022-01-20 18:33:55.123600: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.263311 140191300101952 af.py:224] 2022-01-20 18:33:55.123607: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263353 140191300101952 af.py:224] 2022-01-20 18:33:55.123612: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.263394 140191300101952 af.py:224] 2022-01-20 18:33:55.123620: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.263435 140191300101952 af.py:224] 2022-01-20 18:33:55.123627: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263475 140191300101952 af.py:224] 2022-01-20 18:33:55.123633: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263516 140191300101952 af.py:224] 2022-01-20 18:33:55.123639: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.263557 140191300101952 af.py:224] 2022-01-20 18:33:55.123645: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263597 140191300101952 af.py:224] 2022-01-20 18:33:55.123650: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.263637 140191300101952 af.py:224] 2022-01-20 18:33:55.123658: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.263678 140191300101952 af.py:224] 2022-01-20 18:33:55.123665: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263718 140191300101952 af.py:224] 2022-01-20 18:33:55.123671: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263759 140191300101952 af.py:224] 2022-01-20 18:33:55.123676: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.263799 140191300101952 af.py:224] 2022-01-20 18:33:55.123682: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263843 140191300101952 af.py:224] 2022-01-20 18:33:55.123688: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.263884 140191300101952 af.py:224] 2022-01-20 18:33:55.123696: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.263925 140191300101952 af.py:224] 2022-01-20 18:33:55.123702: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.263965 140191300101952 af.py:224] 2022-01-20 18:33:55.123709: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264006 140191300101952 af.py:224] 2022-01-20 18:33:55.123714: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.264047 140191300101952 af.py:224] 2022-01-20 18:33:55.123734: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264087 140191300101952 af.py:224] 2022-01-20 18:33:55.123739: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.264172 140191300101952 af.py:224] 2022-01-20 18:33:55.123748: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.264214 140191300101952 af.py:224] 2022-01-20 18:33:55.123755: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264255 140191300101952 af.py:224] 2022-01-20 18:33:55.123761: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264295 140191300101952 af.py:224] 2022-01-20 18:33:55.123780: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.264336 140191300101952 af.py:224] 2022-01-20 18:33:55.123787: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264377 140191300101952 af.py:224] 2022-01-20 18:33:55.123792: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.264417 140191300101952 af.py:224] 2022-01-20 18:33:55.123800: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.264458 140191300101952 af.py:224] 2022-01-20 18:33:55.123806: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264499 140191300101952 af.py:224] 2022-01-20 18:33:55.123813: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264543 140191300101952 af.py:224] 2022-01-20 18:33:55.123818: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.264583 140191300101952 af.py:224] 2022-01-20 18:33:55.123824: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264624 140191300101952 af.py:224] 2022-01-20 18:33:55.123829: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.264664 140191300101952 af.py:224] 2022-01-20 18:33:55.123837: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.264705 140191300101952 af.py:224] 2022-01-20 18:33:55.123843: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264745 140191300101952 af.py:224] 2022-01-20 18:33:55.123849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264786 140191300101952 af.py:224] 2022-01-20 18:33:55.123854: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.264826 140191300101952 af.py:224] 2022-01-20 18:33:55.123860: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.264867 140191300101952 af.py:224] 2022-01-20 18:33:55.123865: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.264907 140191300101952 af.py:224] 2022-01-20 18:33:55.123874: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.264961 140191300101952 af.py:224] 2022-01-20 18:33:55.123880: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265002 140191300101952 af.py:224] 2022-01-20 18:33:55.123887: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265043 140191300101952 af.py:224] 2022-01-20 18:33:55.123891: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.265083 140191300101952 af.py:224] 2022-01-20 18:33:55.123897: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265123 140191300101952 af.py:224] 2022-01-20 18:33:55.123902: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.265164 140191300101952 af.py:224] 2022-01-20 18:33:55.123910: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.265204 140191300101952 af.py:224] 2022-01-20 18:33:55.123918: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265248 140191300101952 af.py:224] 2022-01-20 18:33:55.123924: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265289 140191300101952 af.py:224] 2022-01-20 18:33:55.123928: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.265329 140191300101952 af.py:224] 2022-01-20 18:33:55.123934: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265369 140191300101952 af.py:224] 2022-01-20 18:33:55.123939: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.265410 140191300101952 af.py:224] 2022-01-20 18:33:55.123946: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.265450 140191300101952 af.py:224] 2022-01-20 18:33:55.123953: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265491 140191300101952 af.py:224] 2022-01-20 18:33:55.123959: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265531 140191300101952 af.py:224] 2022-01-20 18:33:55.123964: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.265571 140191300101952 af.py:224] 2022-01-20 18:33:55.123970: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265611 140191300101952 af.py:224] 2022-01-20 18:33:55.123975: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.265652 140191300101952 af.py:224] 2022-01-20 18:33:55.123982: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.265692 140191300101952 af.py:224] 2022-01-20 18:33:55.123988: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265733 140191300101952 af.py:224] 2022-01-20 18:33:55.123994: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265774 140191300101952 af.py:224] 2022-01-20 18:33:55.124000: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.265814 140191300101952 af.py:224] 2022-01-20 18:33:55.124005: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265855 140191300101952 af.py:224] 2022-01-20 18:33:55.124010: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.265895 140191300101952 af.py:224] 2022-01-20 18:33:55.124018: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.265943 140191300101952 af.py:224] 2022-01-20 18:33:55.124025: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.265984 140191300101952 af.py:224] 2022-01-20 18:33:55.124032: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266024 140191300101952 af.py:224] 2022-01-20 18:33:55.124037: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.266064 140191300101952 af.py:224] 2022-01-20 18:33:55.124042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266105 140191300101952 af.py:224] 2022-01-20 18:33:55.124047: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.266145 140191300101952 af.py:224] 2022-01-20 18:33:55.124055: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.266186 140191300101952 af.py:224] 2022-01-20 18:33:55.124061: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266227 140191300101952 af.py:224] 2022-01-20 18:33:55.124067: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266267 140191300101952 af.py:224] 2022-01-20 18:33:55.124073: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.266307 140191300101952 af.py:224] 2022-01-20 18:33:55.124080: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266348 140191300101952 af.py:224] 2022-01-20 18:33:55.124085: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.266388 140191300101952 af.py:224] 2022-01-20 18:33:55.124093: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.266428 140191300101952 af.py:224] 2022-01-20 18:33:55.124101: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266469 140191300101952 af.py:224] 2022-01-20 18:33:55.124107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266509 140191300101952 af.py:224] 2022-01-20 18:33:55.124111: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.266549 140191300101952 af.py:224] 2022-01-20 18:33:55.124123: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266605 140191300101952 af.py:224] 2022-01-20 18:33:55.124129: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.266649 140191300101952 af.py:224] 2022-01-20 18:33:55.124137: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.266690 140191300101952 af.py:224] 2022-01-20 18:33:55.124144: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266731 140191300101952 af.py:224] 2022-01-20 18:33:55.124151: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266772 140191300101952 af.py:224] 2022-01-20 18:33:55.124170: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.266812 140191300101952 af.py:224] 2022-01-20 18:33:55.124177: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266853 140191300101952 af.py:224] 2022-01-20 18:33:55.124182: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.266893 140191300101952 af.py:224] 2022-01-20 18:33:55.124189: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.266934 140191300101952 af.py:224] 2022-01-20 18:33:55.124196: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.266975 140191300101952 af.py:224] 2022-01-20 18:33:55.124201: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267016 140191300101952 af.py:224] 2022-01-20 18:33:55.124206: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.267057 140191300101952 af.py:224] 2022-01-20 18:33:55.124212: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267098 140191300101952 af.py:224] 2022-01-20 18:33:55.124217: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.267138 140191300101952 af.py:224] 2022-01-20 18:33:55.124224: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.267179 140191300101952 af.py:224] 2022-01-20 18:33:55.124231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267220 140191300101952 af.py:224] 2022-01-20 18:33:55.124236: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267261 140191300101952 af.py:224] 2022-01-20 18:33:55.124240: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.267302 140191300101952 af.py:224] 2022-01-20 18:33:55.124247: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267346 140191300101952 af.py:224] 2022-01-20 18:33:55.124251: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.267386 140191300101952 af.py:224] 2022-01-20 18:33:55.124259: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.267427 140191300101952 af.py:224] 2022-01-20 18:33:55.124265: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267468 140191300101952 af.py:224] 2022-01-20 18:33:55.124271: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267509 140191300101952 af.py:224] 2022-01-20 18:33:55.124276: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.267550 140191300101952 af.py:224] 2022-01-20 18:33:55.124282: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267590 140191300101952 af.py:224] 2022-01-20 18:33:55.124287: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.267631 140191300101952 af.py:224] 2022-01-20 18:33:55.124294: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.267671 140191300101952 af.py:224] 2022-01-20 18:33:55.124302: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267712 140191300101952 af.py:224] 2022-01-20 18:33:55.124308: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267753 140191300101952 af.py:224] 2022-01-20 18:33:55.124313: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.267793 140191300101952 af.py:224] 2022-01-20 18:33:55.124318: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267834 140191300101952 af.py:224] 2022-01-20 18:33:55.124323: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.267874 140191300101952 af.py:224] 2022-01-20 18:33:55.124331: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.267915 140191300101952 af.py:224] 2022-01-20 18:33:55.124338: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267955 140191300101952 af.py:224] 2022-01-20 18:33:55.124344: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.267998 140191300101952 af.py:224] 2022-01-20 18:33:55.124349: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.268039 140191300101952 af.py:224] 2022-01-20 18:33:55.124355: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.268080 140191300101952 af.py:224] 2022-01-20 18:33:55.124360: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.268125 140191300101952 af.py:224] 2022-01-20 18:33:55.124368: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.268205 140191300101952 af.py:224] 2022-01-20 18:33:55.124374: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.268261 140191300101952 af.py:224] 2022-01-20 18:33:55.124380: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.268304 140191300101952 af.py:224] 2022-01-20 18:33:55.124384: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.268345 140191300101952 af.py:224] 2022-01-20 18:33:55.124390: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.268418 140191300101952 af.py:224] 2022-01-20 18:33:55.124395: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.268493 140191300101952 af.py:224] 2022-01-20 18:33:55.124403: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.268567 140191300101952 af.py:224] 2022-01-20 18:33:55.124409: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.268628 140191300101952 af.py:224] 2022-01-20 18:33:55.124414: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.268684 140191300101952 af.py:224] 2022-01-20 18:33:55.124419: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.268724 140191300101952 af.py:224] 2022-01-20 18:33:55.124425: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.268765 140191300101952 af.py:224] 2022-01-20 18:33:55.124430: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.268805 140191300101952 af.py:224] 2022-01-20 18:33:55.124437: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.268846 140191300101952 af.py:224] 2022-01-20 18:33:55.124443: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.268891 140191300101952 af.py:224] 2022-01-20 18:33:55.124449: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.268931 140191300101952 af.py:224] 2022-01-20 18:33:55.124453: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.268972 140191300101952 af.py:224] 2022-01-20 18:33:55.124460: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269013 140191300101952 af.py:224] 2022-01-20 18:33:55.124465: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.269053 140191300101952 af.py:224] 2022-01-20 18:33:55.124472: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.269094 140191300101952 af.py:224] 2022-01-20 18:33:55.124479: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269134 140191300101952 af.py:224] 2022-01-20 18:33:55.124484: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269175 140191300101952 af.py:224] 2022-01-20 18:33:55.124488: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.269216 140191300101952 af.py:224] 2022-01-20 18:33:55.124494: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269257 140191300101952 af.py:224] 2022-01-20 18:33:55.124498: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.269297 140191300101952 af.py:224] 2022-01-20 18:33:55.124507: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.269337 140191300101952 af.py:224] 2022-01-20 18:33:55.124514: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269378 140191300101952 af.py:224] 2022-01-20 18:33:55.124521: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269419 140191300101952 af.py:224] 2022-01-20 18:33:55.124525: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.269459 140191300101952 af.py:224] 2022-01-20 18:33:55.124531: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269500 140191300101952 af.py:224] 2022-01-20 18:33:55.124536: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.269541 140191300101952 af.py:224] 2022-01-20 18:33:55.124543: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.269585 140191300101952 af.py:224] 2022-01-20 18:33:55.124550: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269626 140191300101952 af.py:224] 2022-01-20 18:33:55.124556: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269667 140191300101952 af.py:224] 2022-01-20 18:33:55.124560: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.269708 140191300101952 af.py:224] 2022-01-20 18:33:55.124566: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269749 140191300101952 af.py:224] 2022-01-20 18:33:55.124570: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.269790 140191300101952 af.py:224] 2022-01-20 18:33:55.124579: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.269831 140191300101952 af.py:224] 2022-01-20 18:33:55.124585: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269871 140191300101952 af.py:224] 2022-01-20 18:33:55.124592: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.269912 140191300101952 af.py:224] 2022-01-20 18:33:55.124596: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.269953 140191300101952 af.py:224] 2022-01-20 18:33:55.124602: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270007 140191300101952 af.py:224] 2022-01-20 18:33:55.124607: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.270048 140191300101952 af.py:224] 2022-01-20 18:33:55.124615: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.270089 140191300101952 af.py:224] 2022-01-20 18:33:55.124621: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270130 140191300101952 af.py:224] 2022-01-20 18:33:55.124627: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270171 140191300101952 af.py:224] 2022-01-20 18:33:55.124631: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.270212 140191300101952 af.py:224] 2022-01-20 18:33:55.124637: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270252 140191300101952 af.py:224] 2022-01-20 18:33:55.124641: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.270296 140191300101952 af.py:224] 2022-01-20 18:33:55.124650: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.270337 140191300101952 af.py:224] 2022-01-20 18:33:55.124657: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270377 140191300101952 af.py:224] 2022-01-20 18:33:55.124663: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270418 140191300101952 af.py:224] 2022-01-20 18:33:55.124667: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.270458 140191300101952 af.py:224] 2022-01-20 18:33:55.124673: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270499 140191300101952 af.py:224] 2022-01-20 18:33:55.124678: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.270540 140191300101952 af.py:224] 2022-01-20 18:33:55.124686: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.270581 140191300101952 af.py:224] 2022-01-20 18:33:55.124692: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270622 140191300101952 af.py:224] 2022-01-20 18:33:55.124698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270663 140191300101952 af.py:224] 2022-01-20 18:33:55.124702: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.270704 140191300101952 af.py:224] 2022-01-20 18:33:55.124708: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270745 140191300101952 af.py:224] 2022-01-20 18:33:55.124713: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.270785 140191300101952 af.py:224] 2022-01-20 18:33:55.124721: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.270826 140191300101952 af.py:224] 2022-01-20 18:33:55.124727: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270866 140191300101952 af.py:224] 2022-01-20 18:33:55.124733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270907 140191300101952 af.py:224] 2022-01-20 18:33:55.124737: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.270948 140191300101952 af.py:224] 2022-01-20 18:33:55.124744: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.270993 140191300101952 af.py:224] 2022-01-20 18:33:55.124749: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.271034 140191300101952 af.py:224] 2022-01-20 18:33:55.124756: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.271074 140191300101952 af.py:224] 2022-01-20 18:33:55.124763: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271115 140191300101952 af.py:224] 2022-01-20 18:33:55.124769: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271156 140191300101952 af.py:224] 2022-01-20 18:33:55.124773: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.271196 140191300101952 af.py:224] 2022-01-20 18:33:55.124779: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271236 140191300101952 af.py:224] 2022-01-20 18:33:55.124783: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.271277 140191300101952 af.py:224] 2022-01-20 18:33:55.124792: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.271317 140191300101952 af.py:224] 2022-01-20 18:33:55.124799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271358 140191300101952 af.py:224] 2022-01-20 18:33:55.124805: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271399 140191300101952 af.py:224] 2022-01-20 18:33:55.124809: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.271439 140191300101952 af.py:224] 2022-01-20 18:33:55.124815: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271479 140191300101952 af.py:224] 2022-01-20 18:33:55.124821: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.271520 140191300101952 af.py:224] 2022-01-20 18:33:55.124828: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.271560 140191300101952 af.py:224] 2022-01-20 18:33:55.124835: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271614 140191300101952 af.py:224] 2022-01-20 18:33:55.124840: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271659 140191300101952 af.py:224] 2022-01-20 18:33:55.124845: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.271701 140191300101952 af.py:224] 2022-01-20 18:33:55.124850: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271741 140191300101952 af.py:224] 2022-01-20 18:33:55.124855: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.271781 140191300101952 af.py:224] 2022-01-20 18:33:55.124862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.271822 140191300101952 af.py:224] 2022-01-20 18:33:55.124869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271863 140191300101952 af.py:224] 2022-01-20 18:33:55.124875: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271904 140191300101952 af.py:224] 2022-01-20 18:33:55.124879: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.271944 140191300101952 af.py:224] 2022-01-20 18:33:55.124885: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.271985 140191300101952 af.py:224] 2022-01-20 18:33:55.124890: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.272026 140191300101952 af.py:224] 2022-01-20 18:33:55.124897: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.272066 140191300101952 af.py:224] 2022-01-20 18:33:55.124905: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272107 140191300101952 af.py:224] 2022-01-20 18:33:55.124912: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272151 140191300101952 af.py:224] 2022-01-20 18:33:55.124916: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.272192 140191300101952 af.py:224] 2022-01-20 18:33:55.124922: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272233 140191300101952 af.py:224] 2022-01-20 18:33:55.124926: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.272274 140191300101952 af.py:224] 2022-01-20 18:33:55.124934: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.272315 140191300101952 af.py:224] 2022-01-20 18:33:55.124941: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272359 140191300101952 af.py:224] 2022-01-20 18:33:55.124947: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272400 140191300101952 af.py:224] 2022-01-20 18:33:55.124951: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.272441 140191300101952 af.py:224] 2022-01-20 18:33:55.124957: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272482 140191300101952 af.py:224] 2022-01-20 18:33:55.124961: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.272522 140191300101952 af.py:224] 2022-01-20 18:33:55.124968: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.272563 140191300101952 af.py:224] 2022-01-20 18:33:55.124975: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272603 140191300101952 af.py:224] 2022-01-20 18:33:55.124980: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272644 140191300101952 af.py:224] 2022-01-20 18:33:55.124984: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.272684 140191300101952 af.py:224] 2022-01-20 18:33:55.124991: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272724 140191300101952 af.py:224] 2022-01-20 18:33:55.124995: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.272764 140191300101952 af.py:224] 2022-01-20 18:33:55.125003: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.272804 140191300101952 af.py:224] 2022-01-20 18:33:55.125011: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272845 140191300101952 af.py:224] 2022-01-20 18:33:55.125017: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272886 140191300101952 af.py:224] 2022-01-20 18:33:55.125022: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.272926 140191300101952 af.py:224] 2022-01-20 18:33:55.125028: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.272967 140191300101952 af.py:224] 2022-01-20 18:33:55.125033: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.273007 140191300101952 af.py:224] 2022-01-20 18:33:55.125040: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.273051 140191300101952 af.py:224] 2022-01-20 18:33:55.125047: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273091 140191300101952 af.py:224] 2022-01-20 18:33:55.125053: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273132 140191300101952 af.py:224] 2022-01-20 18:33:55.125057: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.273173 140191300101952 af.py:224] 2022-01-20 18:33:55.125064: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273213 140191300101952 af.py:224] 2022-01-20 18:33:55.125068: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.273267 140191300101952 af.py:224] 2022-01-20 18:33:55.125076: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.273308 140191300101952 af.py:224] 2022-01-20 18:33:55.125082: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273349 140191300101952 af.py:224] 2022-01-20 18:33:55.125088: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273390 140191300101952 af.py:224] 2022-01-20 18:33:55.125092: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.273431 140191300101952 af.py:224] 2022-01-20 18:33:55.125098: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273471 140191300101952 af.py:224] 2022-01-20 18:33:55.125103: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.273512 140191300101952 af.py:224] 2022-01-20 18:33:55.125111: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.273553 140191300101952 af.py:224] 2022-01-20 18:33:55.125118: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273594 140191300101952 af.py:224] 2022-01-20 18:33:55.125123: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273635 140191300101952 af.py:224] 2022-01-20 18:33:55.125127: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.273676 140191300101952 af.py:224] 2022-01-20 18:33:55.125134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273717 140191300101952 af.py:224] 2022-01-20 18:33:55.125138: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.273761 140191300101952 af.py:224] 2022-01-20 18:33:55.125146: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.273802 140191300101952 af.py:224] 2022-01-20 18:33:55.125152: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273843 140191300101952 af.py:224] 2022-01-20 18:33:55.125158: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273883 140191300101952 af.py:224] 2022-01-20 18:33:55.125162: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.273924 140191300101952 af.py:224] 2022-01-20 18:33:55.125169: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.273965 140191300101952 af.py:224] 2022-01-20 18:33:55.125187: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.274006 140191300101952 af.py:224] 2022-01-20 18:33:55.125196: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.274046 140191300101952 af.py:224] 2022-01-20 18:33:55.125203: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274087 140191300101952 af.py:224] 2022-01-20 18:33:55.125210: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274127 140191300101952 af.py:224] 2022-01-20 18:33:55.125214: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.274168 140191300101952 af.py:224] 2022-01-20 18:33:55.125220: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274209 140191300101952 af.py:224] 2022-01-20 18:33:55.125224: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.274250 140191300101952 af.py:224] 2022-01-20 18:33:55.125233: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.274291 140191300101952 af.py:224] 2022-01-20 18:33:55.125239: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274331 140191300101952 af.py:224] 2022-01-20 18:33:55.125245: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274372 140191300101952 af.py:224] 2022-01-20 18:33:55.125249: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.274413 140191300101952 af.py:224] 2022-01-20 18:33:55.125256: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274456 140191300101952 af.py:224] 2022-01-20 18:33:55.125260: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.274497 140191300101952 af.py:224] 2022-01-20 18:33:55.125268: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.274538 140191300101952 af.py:224] 2022-01-20 18:33:55.125275: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274579 140191300101952 af.py:224] 2022-01-20 18:33:55.125280: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274619 140191300101952 af.py:224] 2022-01-20 18:33:55.125284: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.274659 140191300101952 af.py:224] 2022-01-20 18:33:55.125291: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274700 140191300101952 af.py:224] 2022-01-20 18:33:55.125296: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.274741 140191300101952 af.py:224] 2022-01-20 18:33:55.125303: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.274781 140191300101952 af.py:224] 2022-01-20 18:33:55.125309: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274837 140191300101952 af.py:224] 2022-01-20 18:33:55.125316: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274897 140191300101952 af.py:224] 2022-01-20 18:33:55.125322: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.274938 140191300101952 af.py:224] 2022-01-20 18:33:55.125328: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.274979 140191300101952 af.py:224] 2022-01-20 18:33:55.125332: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.275037 140191300101952 af.py:224] 2022-01-20 18:33:55.125340: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.275078 140191300101952 af.py:224] 2022-01-20 18:33:55.125346: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275119 140191300101952 af.py:224] 2022-01-20 18:33:55.125352: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275163 140191300101952 af.py:224] 2022-01-20 18:33:55.125356: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.275205 140191300101952 af.py:224] 2022-01-20 18:33:55.125362: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275246 140191300101952 af.py:224] 2022-01-20 18:33:55.125367: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.275286 140191300101952 af.py:224] 2022-01-20 18:33:55.125375: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.275327 140191300101952 af.py:224] 2022-01-20 18:33:55.125382: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275367 140191300101952 af.py:224] 2022-01-20 18:33:55.125389: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275408 140191300101952 af.py:224] 2022-01-20 18:33:55.125393: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.275448 140191300101952 af.py:224] 2022-01-20 18:33:55.125399: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275489 140191300101952 af.py:224] 2022-01-20 18:33:55.125403: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.275530 140191300101952 af.py:224] 2022-01-20 18:33:55.125411: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.275571 140191300101952 af.py:224] 2022-01-20 18:33:55.125417: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275611 140191300101952 af.py:224] 2022-01-20 18:33:55.125423: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275652 140191300101952 af.py:224] 2022-01-20 18:33:55.125427: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.275692 140191300101952 af.py:224] 2022-01-20 18:33:55.125433: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275733 140191300101952 af.py:224] 2022-01-20 18:33:55.125438: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.275774 140191300101952 af.py:224] 2022-01-20 18:33:55.125446: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.275815 140191300101952 af.py:224] 2022-01-20 18:33:55.125452: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275858 140191300101952 af.py:224] 2022-01-20 18:33:55.125458: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275900 140191300101952 af.py:224] 2022-01-20 18:33:55.125462: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.275940 140191300101952 af.py:224] 2022-01-20 18:33:55.125468: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.275981 140191300101952 af.py:224] 2022-01-20 18:33:55.125473: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.276021 140191300101952 af.py:224] 2022-01-20 18:33:55.125481: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.276062 140191300101952 af.py:224] 2022-01-20 18:33:55.125487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276103 140191300101952 af.py:224] 2022-01-20 18:33:55.125493: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276148 140191300101952 af.py:224] 2022-01-20 18:33:55.125497: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.276189 140191300101952 af.py:224] 2022-01-20 18:33:55.125504: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276229 140191300101952 af.py:224] 2022-01-20 18:33:55.125508: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.276270 140191300101952 af.py:224] 2022-01-20 18:33:55.125516: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.276310 140191300101952 af.py:224] 2022-01-20 18:33:55.125522: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276350 140191300101952 af.py:224] 2022-01-20 18:33:55.125527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276391 140191300101952 af.py:224] 2022-01-20 18:33:55.125532: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.276431 140191300101952 af.py:224] 2022-01-20 18:33:55.125538: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276472 140191300101952 af.py:224] 2022-01-20 18:33:55.125542: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.276526 140191300101952 af.py:224] 2022-01-20 18:33:55.125550: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.276571 140191300101952 af.py:224] 2022-01-20 18:33:55.125557: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276612 140191300101952 af.py:224] 2022-01-20 18:33:55.125563: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276652 140191300101952 af.py:224] 2022-01-20 18:33:55.125567: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.276693 140191300101952 af.py:224] 2022-01-20 18:33:55.125573: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276733 140191300101952 af.py:224] 2022-01-20 18:33:55.125578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.276773 140191300101952 af.py:224] 2022-01-20 18:33:55.125586: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.276813 140191300101952 af.py:224] 2022-01-20 18:33:55.125592: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276854 140191300101952 af.py:224] 2022-01-20 18:33:55.125598: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276895 140191300101952 af.py:224] 2022-01-20 18:33:55.125602: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.276934 140191300101952 af.py:224] 2022-01-20 18:33:55.125609: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.276975 140191300101952 af.py:224] 2022-01-20 18:33:55.125613: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.277015 140191300101952 af.py:224] 2022-01-20 18:33:55.125621: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.277056 140191300101952 af.py:224] 2022-01-20 18:33:55.125628: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277096 140191300101952 af.py:224] 2022-01-20 18:33:55.125634: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277136 140191300101952 af.py:224] 2022-01-20 18:33:55.125638: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.277177 140191300101952 af.py:224] 2022-01-20 18:33:55.125644: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277217 140191300101952 af.py:224] 2022-01-20 18:33:55.125648: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.277261 140191300101952 af.py:224] 2022-01-20 18:33:55.125656: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.277302 140191300101952 af.py:224] 2022-01-20 18:33:55.125662: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277343 140191300101952 af.py:224] 2022-01-20 18:33:55.125669: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277384 140191300101952 af.py:224] 2022-01-20 18:33:55.125673: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.277426 140191300101952 af.py:224] 2022-01-20 18:33:55.125679: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277466 140191300101952 af.py:224] 2022-01-20 18:33:55.125683: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.277507 140191300101952 af.py:224] 2022-01-20 18:33:55.125690: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.277548 140191300101952 af.py:224] 2022-01-20 18:33:55.125697: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277589 140191300101952 af.py:224] 2022-01-20 18:33:55.125703: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277630 140191300101952 af.py:224] 2022-01-20 18:33:55.125708: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.277670 140191300101952 af.py:224] 2022-01-20 18:33:55.125714: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277710 140191300101952 af.py:224] 2022-01-20 18:33:55.125719: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.277751 140191300101952 af.py:224] 2022-01-20 18:33:55.125726: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.277791 140191300101952 af.py:224] 2022-01-20 18:33:55.125733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277832 140191300101952 af.py:224] 2022-01-20 18:33:55.125740: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277872 140191300101952 af.py:224] 2022-01-20 18:33:55.125744: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.277913 140191300101952 af.py:224] 2022-01-20 18:33:55.125750: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.277957 140191300101952 af.py:224] 2022-01-20 18:33:55.125754: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.277997 140191300101952 af.py:224] 2022-01-20 18:33:55.125761: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.278038 140191300101952 af.py:224] 2022-01-20 18:33:55.125769: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278079 140191300101952 af.py:224] 2022-01-20 18:33:55.125776: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278133 140191300101952 af.py:224] 2022-01-20 18:33:55.125782: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.278175 140191300101952 af.py:224] 2022-01-20 18:33:55.125787: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278216 140191300101952 af.py:224] 2022-01-20 18:33:55.125792: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.278256 140191300101952 af.py:224] 2022-01-20 18:33:55.125800: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.278297 140191300101952 af.py:224] 2022-01-20 18:33:55.125806: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278337 140191300101952 af.py:224] 2022-01-20 18:33:55.125813: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278378 140191300101952 af.py:224] 2022-01-20 18:33:55.125817: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.278419 140191300101952 af.py:224] 2022-01-20 18:33:55.125823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278459 140191300101952 af.py:224] 2022-01-20 18:33:55.125829: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.278500 140191300101952 af.py:224] 2022-01-20 18:33:55.125836: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.278540 140191300101952 af.py:224] 2022-01-20 18:33:55.125843: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278581 140191300101952 af.py:224] 2022-01-20 18:33:55.125850: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278625 140191300101952 af.py:224] 2022-01-20 18:33:55.125854: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.278665 140191300101952 af.py:224] 2022-01-20 18:33:55.125861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278706 140191300101952 af.py:224] 2022-01-20 18:33:55.125865: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.278747 140191300101952 af.py:224] 2022-01-20 18:33:55.125873: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.278787 140191300101952 af.py:224] 2022-01-20 18:33:55.125879: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278828 140191300101952 af.py:224] 2022-01-20 18:33:55.125886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278868 140191300101952 af.py:224] 2022-01-20 18:33:55.125891: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.278908 140191300101952 af.py:224] 2022-01-20 18:33:55.125896: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.278949 140191300101952 af.py:224] 2022-01-20 18:33:55.125901: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.278989 140191300101952 af.py:224] 2022-01-20 18:33:55.125909: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.279029 140191300101952 af.py:224] 2022-01-20 18:33:55.125915: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279069 140191300101952 af.py:224] 2022-01-20 18:33:55.125921: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279110 140191300101952 af.py:224] 2022-01-20 18:33:55.125926: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.279151 140191300101952 af.py:224] 2022-01-20 18:33:55.125931: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279191 140191300101952 af.py:224] 2022-01-20 18:33:55.125936: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.279232 140191300101952 af.py:224] 2022-01-20 18:33:55.125944: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.279272 140191300101952 af.py:224] 2022-01-20 18:33:55.125950: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279316 140191300101952 af.py:224] 2022-01-20 18:33:55.125956: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279356 140191300101952 af.py:224] 2022-01-20 18:33:55.125961: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.279396 140191300101952 af.py:224] 2022-01-20 18:33:55.125967: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279437 140191300101952 af.py:224] 2022-01-20 18:33:55.125972: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.279478 140191300101952 af.py:224] 2022-01-20 18:33:55.125980: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.279518 140191300101952 af.py:224] 2022-01-20 18:33:55.125987: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279558 140191300101952 af.py:224] 2022-01-20 18:33:55.125993: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279599 140191300101952 af.py:224] 2022-01-20 18:33:55.125997: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.279640 140191300101952 af.py:224] 2022-01-20 18:33:55.126004: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279680 140191300101952 af.py:224] 2022-01-20 18:33:55.126009: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.279734 140191300101952 af.py:224] 2022-01-20 18:33:55.126016: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.279776 140191300101952 af.py:224] 2022-01-20 18:33:55.126023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279816 140191300101952 af.py:224] 2022-01-20 18:33:55.126029: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279857 140191300101952 af.py:224] 2022-01-20 18:33:55.126034: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.279898 140191300101952 af.py:224] 2022-01-20 18:33:55.126039: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.279939 140191300101952 af.py:224] 2022-01-20 18:33:55.126045: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.279980 140191300101952 af.py:224] 2022-01-20 18:33:55.126052: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.280024 140191300101952 af.py:224] 2022-01-20 18:33:55.126059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280065 140191300101952 af.py:224] 2022-01-20 18:33:55.126066: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280106 140191300101952 af.py:224] 2022-01-20 18:33:55.126071: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.280150 140191300101952 af.py:224] 2022-01-20 18:33:55.126078: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280191 140191300101952 af.py:224] 2022-01-20 18:33:55.126083: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.280232 140191300101952 af.py:224] 2022-01-20 18:33:55.126091: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.280272 140191300101952 af.py:224] 2022-01-20 18:33:55.126098: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280313 140191300101952 af.py:224] 2022-01-20 18:33:55.126105: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280354 140191300101952 af.py:224] 2022-01-20 18:33:55.126110: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.280395 140191300101952 af.py:224] 2022-01-20 18:33:55.126116: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280436 140191300101952 af.py:224] 2022-01-20 18:33:55.126121: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.280477 140191300101952 af.py:224] 2022-01-20 18:33:55.126129: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.280517 140191300101952 af.py:224] 2022-01-20 18:33:55.126136: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280558 140191300101952 af.py:224] 2022-01-20 18:33:55.126142: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280598 140191300101952 af.py:224] 2022-01-20 18:33:55.126147: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.280638 140191300101952 af.py:224] 2022-01-20 18:33:55.126153: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280679 140191300101952 af.py:224] 2022-01-20 18:33:55.126158: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.280723 140191300101952 af.py:224] 2022-01-20 18:33:55.126165: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.280764 140191300101952 af.py:224] 2022-01-20 18:33:55.126171: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280805 140191300101952 af.py:224] 2022-01-20 18:33:55.126178: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280846 140191300101952 af.py:224] 2022-01-20 18:33:55.126182: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.280887 140191300101952 af.py:224] 2022-01-20 18:33:55.126189: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.280927 140191300101952 af.py:224] 2022-01-20 18:33:55.126193: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.280968 140191300101952 af.py:224] 2022-01-20 18:33:55.126201: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.281009 140191300101952 af.py:224] 2022-01-20 18:33:55.126207: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281049 140191300101952 af.py:224] 2022-01-20 18:33:55.126213: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281090 140191300101952 af.py:224] 2022-01-20 18:33:55.126218: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.281131 140191300101952 af.py:224] 2022-01-20 18:33:55.126224: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281171 140191300101952 af.py:224] 2022-01-20 18:33:55.126229: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.281211 140191300101952 af.py:224] 2022-01-20 18:33:55.126237: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.281252 140191300101952 af.py:224] 2022-01-20 18:33:55.126243: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281293 140191300101952 af.py:224] 2022-01-20 18:33:55.126249: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281351 140191300101952 af.py:224] 2022-01-20 18:33:55.126253: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.281399 140191300101952 af.py:224] 2022-01-20 18:33:55.126260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281458 140191300101952 af.py:224] 2022-01-20 18:33:55.126265: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.281499 140191300101952 af.py:224] 2022-01-20 18:33:55.126272: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.281541 140191300101952 af.py:224] 2022-01-20 18:33:55.126279: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281582 140191300101952 af.py:224] 2022-01-20 18:33:55.126285: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281624 140191300101952 af.py:224] 2022-01-20 18:33:55.126290: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.281680 140191300101952 af.py:224] 2022-01-20 18:33:55.126296: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281721 140191300101952 af.py:224] 2022-01-20 18:33:55.126301: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.281761 140191300101952 af.py:224] 2022-01-20 18:33:55.126309: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.281801 140191300101952 af.py:224] 2022-01-20 18:33:55.126315: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281841 140191300101952 af.py:224] 2022-01-20 18:33:55.126321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281882 140191300101952 af.py:224] 2022-01-20 18:33:55.126325: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.281923 140191300101952 af.py:224] 2022-01-20 18:33:55.126331: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.281964 140191300101952 af.py:224] 2022-01-20 18:33:55.126336: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.282005 140191300101952 af.py:224] 2022-01-20 18:33:55.126343: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.282045 140191300101952 af.py:224] 2022-01-20 18:33:55.126350: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282086 140191300101952 af.py:224] 2022-01-20 18:33:55.126357: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282130 140191300101952 af.py:224] 2022-01-20 18:33:55.126361: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.282170 140191300101952 af.py:224] 2022-01-20 18:33:55.126367: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282211 140191300101952 af.py:224] 2022-01-20 18:33:55.126372: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.282251 140191300101952 af.py:224] 2022-01-20 18:33:55.126380: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.282292 140191300101952 af.py:224] 2022-01-20 18:33:55.126387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282332 140191300101952 af.py:224] 2022-01-20 18:33:55.126393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282374 140191300101952 af.py:224] 2022-01-20 18:33:55.126398: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.282414 140191300101952 af.py:224] 2022-01-20 18:33:55.126405: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282455 140191300101952 af.py:224] 2022-01-20 18:33:55.126410: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.282495 140191300101952 af.py:224] 2022-01-20 18:33:55.126418: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.282536 140191300101952 af.py:224] 2022-01-20 18:33:55.126424: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282577 140191300101952 af.py:224] 2022-01-20 18:33:55.126430: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282617 140191300101952 af.py:224] 2022-01-20 18:33:55.126435: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.282658 140191300101952 af.py:224] 2022-01-20 18:33:55.126441: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282699 140191300101952 af.py:224] 2022-01-20 18:33:55.126446: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.282740 140191300101952 af.py:224] 2022-01-20 18:33:55.126454: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.282781 140191300101952 af.py:224] 2022-01-20 18:33:55.126461: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282824 140191300101952 af.py:224] 2022-01-20 18:33:55.126467: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282866 140191300101952 af.py:224] 2022-01-20 18:33:55.126472: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.282906 140191300101952 af.py:224] 2022-01-20 18:33:55.126478: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.282948 140191300101952 af.py:224] 2022-01-20 18:33:55.126483: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.282988 140191300101952 af.py:224] 2022-01-20 18:33:55.126491: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.283041 140191300101952 af.py:224] 2022-01-20 18:33:55.126498: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283083 140191300101952 af.py:224] 2022-01-20 18:33:55.126504: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283124 140191300101952 af.py:224] 2022-01-20 18:33:55.126509: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.283165 140191300101952 af.py:224] 2022-01-20 18:33:55.126514: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283206 140191300101952 af.py:224] 2022-01-20 18:33:55.126519: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.283247 140191300101952 af.py:224] 2022-01-20 18:33:55.126527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.283288 140191300101952 af.py:224] 2022-01-20 18:33:55.126534: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283329 140191300101952 af.py:224] 2022-01-20 18:33:55.126540: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283370 140191300101952 af.py:224] 2022-01-20 18:33:55.126545: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.283411 140191300101952 af.py:224] 2022-01-20 18:33:55.126551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283452 140191300101952 af.py:224] 2022-01-20 18:33:55.126556: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.283493 140191300101952 af.py:224] 2022-01-20 18:33:55.126563: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.283536 140191300101952 af.py:224] 2022-01-20 18:33:55.126570: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283577 140191300101952 af.py:224] 2022-01-20 18:33:55.126576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283617 140191300101952 af.py:224] 2022-01-20 18:33:55.126581: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.283658 140191300101952 af.py:224] 2022-01-20 18:33:55.126587: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283699 140191300101952 af.py:224] 2022-01-20 18:33:55.126591: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.283739 140191300101952 af.py:224] 2022-01-20 18:33:55.126599: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.283779 140191300101952 af.py:224] 2022-01-20 18:33:55.126606: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283820 140191300101952 af.py:224] 2022-01-20 18:33:55.126611: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283860 140191300101952 af.py:224] 2022-01-20 18:33:55.126615: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.283901 140191300101952 af.py:224] 2022-01-20 18:33:55.126622: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.283942 140191300101952 af.py:224] 2022-01-20 18:33:55.126627: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.283982 140191300101952 af.py:224] 2022-01-20 18:33:55.126634: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.284023 140191300101952 af.py:224] 2022-01-20 18:33:55.126641: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284063 140191300101952 af.py:224] 2022-01-20 18:33:55.126647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284103 140191300101952 af.py:224] 2022-01-20 18:33:55.126651: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.284148 140191300101952 af.py:224] 2022-01-20 18:33:55.126658: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284193 140191300101952 af.py:224] 2022-01-20 18:33:55.126663: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.284234 140191300101952 af.py:224] 2022-01-20 18:33:55.126671: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.284275 140191300101952 af.py:224] 2022-01-20 18:33:55.126677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284316 140191300101952 af.py:224] 2022-01-20 18:33:55.126682: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284356 140191300101952 af.py:224] 2022-01-20 18:33:55.126687: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.284397 140191300101952 af.py:224] 2022-01-20 18:33:55.126693: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284438 140191300101952 af.py:224] 2022-01-20 18:33:55.126698: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.284479 140191300101952 af.py:224] 2022-01-20 18:33:55.126706: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.284519 140191300101952 af.py:224] 2022-01-20 18:33:55.126712: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284560 140191300101952 af.py:224] 2022-01-20 18:33:55.126718: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284601 140191300101952 af.py:224] 2022-01-20 18:33:55.126723: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.284655 140191300101952 af.py:224] 2022-01-20 18:33:55.126729: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284697 140191300101952 af.py:224] 2022-01-20 18:33:55.126734: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.284738 140191300101952 af.py:224] 2022-01-20 18:33:55.126742: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.284779 140191300101952 af.py:224] 2022-01-20 18:33:55.126748: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284820 140191300101952 af.py:224] 2022-01-20 18:33:55.126754: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284860 140191300101952 af.py:224] 2022-01-20 18:33:55.126758: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.284905 140191300101952 af.py:224] 2022-01-20 18:33:55.126765: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.284946 140191300101952 af.py:224] 2022-01-20 18:33:55.126769: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.284987 140191300101952 af.py:224] 2022-01-20 18:33:55.126776: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.285028 140191300101952 af.py:224] 2022-01-20 18:33:55.126783: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285069 140191300101952 af.py:224] 2022-01-20 18:33:55.126789: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285110 140191300101952 af.py:224] 2022-01-20 18:33:55.126793: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.285151 140191300101952 af.py:224] 2022-01-20 18:33:55.126799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285191 140191300101952 af.py:224] 2022-01-20 18:33:55.126804: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.285231 140191300101952 af.py:224] 2022-01-20 18:33:55.126812: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.285272 140191300101952 af.py:224] 2022-01-20 18:33:55.126820: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285312 140191300101952 af.py:224] 2022-01-20 18:33:55.126826: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285353 140191300101952 af.py:224] 2022-01-20 18:33:55.126831: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.285393 140191300101952 af.py:224] 2022-01-20 18:33:55.126836: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285433 140191300101952 af.py:224] 2022-01-20 18:33:55.126841: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.285474 140191300101952 af.py:224] 2022-01-20 18:33:55.126848: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.285514 140191300101952 af.py:224] 2022-01-20 18:33:55.126855: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285554 140191300101952 af.py:224] 2022-01-20 18:33:55.126861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285598 140191300101952 af.py:224] 2022-01-20 18:33:55.126865: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.285639 140191300101952 af.py:224] 2022-01-20 18:33:55.126872: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285679 140191300101952 af.py:224] 2022-01-20 18:33:55.126877: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.285720 140191300101952 af.py:224] 2022-01-20 18:33:55.126885: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.285760 140191300101952 af.py:224] 2022-01-20 18:33:55.126891: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285801 140191300101952 af.py:224] 2022-01-20 18:33:55.126897: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285841 140191300101952 af.py:224] 2022-01-20 18:33:55.126902: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.285881 140191300101952 af.py:224] 2022-01-20 18:33:55.126908: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.285922 140191300101952 af.py:224] 2022-01-20 18:33:55.126913: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.285962 140191300101952 af.py:224] 2022-01-20 18:33:55.126920: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.286003 140191300101952 af.py:224] 2022-01-20 18:33:55.126927: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286044 140191300101952 af.py:224] 2022-01-20 18:33:55.126934: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286084 140191300101952 af.py:224] 2022-01-20 18:33:55.126938: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.286125 140191300101952 af.py:224] 2022-01-20 18:33:55.126944: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286166 140191300101952 af.py:224] 2022-01-20 18:33:55.126949: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.286207 140191300101952 af.py:224] 2022-01-20 18:33:55.126957: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.286261 140191300101952 af.py:224] 2022-01-20 18:33:55.126963: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286306 140191300101952 af.py:224] 2022-01-20 18:33:55.126969: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286348 140191300101952 af.py:224] 2022-01-20 18:33:55.126974: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.286388 140191300101952 af.py:224] 2022-01-20 18:33:55.126980: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286429 140191300101952 af.py:224] 2022-01-20 18:33:55.126985: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.286470 140191300101952 af.py:224] 2022-01-20 18:33:55.126992: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.286510 140191300101952 af.py:224] 2022-01-20 18:33:55.126999: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286551 140191300101952 af.py:224] 2022-01-20 18:33:55.127005: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286592 140191300101952 af.py:224] 2022-01-20 18:33:55.127009: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.286633 140191300101952 af.py:224] 2022-01-20 18:33:55.127015: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286673 140191300101952 af.py:224] 2022-01-20 18:33:55.127020: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.286714 140191300101952 af.py:224] 2022-01-20 18:33:55.127028: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.286754 140191300101952 af.py:224] 2022-01-20 18:33:55.127034: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286795 140191300101952 af.py:224] 2022-01-20 18:33:55.127040: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286836 140191300101952 af.py:224] 2022-01-20 18:33:55.127045: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.286876 140191300101952 af.py:224] 2022-01-20 18:33:55.127051: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.286917 140191300101952 af.py:224] 2022-01-20 18:33:55.127056: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.286958 140191300101952 af.py:224] 2022-01-20 18:33:55.127063: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.287002 140191300101952 af.py:224] 2022-01-20 18:33:55.127070: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287043 140191300101952 af.py:224] 2022-01-20 18:33:55.127076: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287084 140191300101952 af.py:224] 2022-01-20 18:33:55.127081: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.287124 140191300101952 af.py:224] 2022-01-20 18:33:55.127088: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287164 140191300101952 af.py:224] 2022-01-20 18:33:55.127093: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.287205 140191300101952 af.py:224] 2022-01-20 18:33:55.127102: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.287246 140191300101952 af.py:224] 2022-01-20 18:33:55.127108: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287286 140191300101952 af.py:224] 2022-01-20 18:33:55.127115: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287328 140191300101952 af.py:224] 2022-01-20 18:33:55.127120: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.287368 140191300101952 af.py:224] 2022-01-20 18:33:55.127127: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287408 140191300101952 af.py:224] 2022-01-20 18:33:55.127131: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.287449 140191300101952 af.py:224] 2022-01-20 18:33:55.127139: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.287490 140191300101952 af.py:224] 2022-01-20 18:33:55.127146: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287531 140191300101952 af.py:224] 2022-01-20 18:33:55.127153: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287571 140191300101952 af.py:224] 2022-01-20 18:33:55.127157: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.287612 140191300101952 af.py:224] 2022-01-20 18:33:55.127163: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287656 140191300101952 af.py:224] 2022-01-20 18:33:55.127168: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.287697 140191300101952 af.py:224] 2022-01-20 18:33:55.127176: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.287738 140191300101952 af.py:224] 2022-01-20 18:33:55.127183: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287779 140191300101952 af.py:224] 2022-01-20 18:33:55.127190: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287819 140191300101952 af.py:224] 2022-01-20 18:33:55.127195: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.287875 140191300101952 af.py:224] 2022-01-20 18:33:55.127201: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.287918 140191300101952 af.py:224] 2022-01-20 18:33:55.127206: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.287976 140191300101952 af.py:224] 2022-01-20 18:33:55.127215: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.288018 140191300101952 af.py:224] 2022-01-20 18:33:55.127222: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288058 140191300101952 af.py:224] 2022-01-20 18:33:55.127229: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288100 140191300101952 af.py:224] 2022-01-20 18:33:55.127234: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.288159 140191300101952 af.py:224] 2022-01-20 18:33:55.127240: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288201 140191300101952 af.py:224] 2022-01-20 18:33:55.127245: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.288241 140191300101952 af.py:224] 2022-01-20 18:33:55.127253: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.288282 140191300101952 af.py:224] 2022-01-20 18:33:55.127259: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288322 140191300101952 af.py:224] 2022-01-20 18:33:55.127266: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288363 140191300101952 af.py:224] 2022-01-20 18:33:55.127271: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.288407 140191300101952 af.py:224] 2022-01-20 18:33:55.127277: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288448 140191300101952 af.py:224] 2022-01-20 18:33:55.127281: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.288489 140191300101952 af.py:224] 2022-01-20 18:33:55.127289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.288530 140191300101952 af.py:224] 2022-01-20 18:33:55.127295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288571 140191300101952 af.py:224] 2022-01-20 18:33:55.127301: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288612 140191300101952 af.py:224] 2022-01-20 18:33:55.127306: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.288653 140191300101952 af.py:224] 2022-01-20 18:33:55.127313: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288693 140191300101952 af.py:224] 2022-01-20 18:33:55.127318: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.288734 140191300101952 af.py:224] 2022-01-20 18:33:55.127326: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.288774 140191300101952 af.py:224] 2022-01-20 18:33:55.127333: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288815 140191300101952 af.py:224] 2022-01-20 18:33:55.127339: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288856 140191300101952 af.py:224] 2022-01-20 18:33:55.127344: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.288896 140191300101952 af.py:224] 2022-01-20 18:33:55.127351: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.288938 140191300101952 af.py:224] 2022-01-20 18:33:55.127356: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.288978 140191300101952 af.py:224] 2022-01-20 18:33:55.127363: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.289019 140191300101952 af.py:224] 2022-01-20 18:33:55.127370: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289059 140191300101952 af.py:224] 2022-01-20 18:33:55.127376: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289103 140191300101952 af.py:224] 2022-01-20 18:33:55.127381: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.289143 140191300101952 af.py:224] 2022-01-20 18:33:55.127387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289184 140191300101952 af.py:224] 2022-01-20 18:33:55.127392: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.289225 140191300101952 af.py:224] 2022-01-20 18:33:55.127400: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.289265 140191300101952 af.py:224] 2022-01-20 18:33:55.127406: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289306 140191300101952 af.py:224] 2022-01-20 18:33:55.127412: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289348 140191300101952 af.py:224] 2022-01-20 18:33:55.127417: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.289389 140191300101952 af.py:224] 2022-01-20 18:33:55.127423: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289429 140191300101952 af.py:224] 2022-01-20 18:33:55.127428: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.289469 140191300101952 af.py:224] 2022-01-20 18:33:55.127437: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.289524 140191300101952 af.py:224] 2022-01-20 18:33:55.127443: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289565 140191300101952 af.py:224] 2022-01-20 18:33:55.127449: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289606 140191300101952 af.py:224] 2022-01-20 18:33:55.127454: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.289646 140191300101952 af.py:224] 2022-01-20 18:33:55.127461: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289687 140191300101952 af.py:224] 2022-01-20 18:33:55.127466: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.289728 140191300101952 af.py:224] 2022-01-20 18:33:55.127474: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.289769 140191300101952 af.py:224] 2022-01-20 18:33:55.127480: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289813 140191300101952 af.py:224] 2022-01-20 18:33:55.127487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289854 140191300101952 af.py:224] 2022-01-20 18:33:55.127492: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.289895 140191300101952 af.py:224] 2022-01-20 18:33:55.127498: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.289936 140191300101952 af.py:224] 2022-01-20 18:33:55.127503: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.289977 140191300101952 af.py:224] 2022-01-20 18:33:55.127510: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.290017 140191300101952 af.py:224] 2022-01-20 18:33:55.127518: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290058 140191300101952 af.py:224] 2022-01-20 18:33:55.127524: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290099 140191300101952 af.py:224] 2022-01-20 18:33:55.127529: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.290140 140191300101952 af.py:224] 2022-01-20 18:33:55.127535: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290181 140191300101952 af.py:224] 2022-01-20 18:33:55.127540: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.290222 140191300101952 af.py:224] 2022-01-20 18:33:55.127547: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.290262 140191300101952 af.py:224] 2022-01-20 18:33:55.127555: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290303 140191300101952 af.py:224] 2022-01-20 18:33:55.127561: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290345 140191300101952 af.py:224] 2022-01-20 18:33:55.127566: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.290385 140191300101952 af.py:224] 2022-01-20 18:33:55.127573: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290425 140191300101952 af.py:224] 2022-01-20 18:33:55.127578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.290466 140191300101952 af.py:224] 2022-01-20 18:33:55.127587: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.290511 140191300101952 af.py:224] 2022-01-20 18:33:55.127594: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290552 140191300101952 af.py:224] 2022-01-20 18:33:55.127600: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290593 140191300101952 af.py:224] 2022-01-20 18:33:55.127605: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.290633 140191300101952 af.py:224] 2022-01-20 18:33:55.127612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290675 140191300101952 af.py:224] 2022-01-20 18:33:55.127617: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.290715 140191300101952 af.py:224] 2022-01-20 18:33:55.127626: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.290756 140191300101952 af.py:224] 2022-01-20 18:33:55.127632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290797 140191300101952 af.py:224] 2022-01-20 18:33:55.127639: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290838 140191300101952 af.py:224] 2022-01-20 18:33:55.127644: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.290879 140191300101952 af.py:224] 2022-01-20 18:33:55.127650: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.290920 140191300101952 af.py:224] 2022-01-20 18:33:55.127655: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.290960 140191300101952 af.py:224] 2022-01-20 18:33:55.127664: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.291000 140191300101952 af.py:224] 2022-01-20 18:33:55.127670: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291041 140191300101952 af.py:224] 2022-01-20 18:33:55.127676: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291082 140191300101952 af.py:224] 2022-01-20 18:33:55.127681: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.291136 140191300101952 af.py:224] 2022-01-20 18:33:55.127688: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291182 140191300101952 af.py:224] 2022-01-20 18:33:55.127693: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.291223 140191300101952 af.py:224] 2022-01-20 18:33:55.127702: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.291264 140191300101952 af.py:224] 2022-01-20 18:33:55.127708: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291305 140191300101952 af.py:224] 2022-01-20 18:33:55.127715: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291345 140191300101952 af.py:224] 2022-01-20 18:33:55.127720: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.291385 140191300101952 af.py:224] 2022-01-20 18:33:55.127726: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291426 140191300101952 af.py:224] 2022-01-20 18:33:55.127731: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.291467 140191300101952 af.py:224] 2022-01-20 18:33:55.127739: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.291508 140191300101952 af.py:224] 2022-01-20 18:33:55.127746: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291549 140191300101952 af.py:224] 2022-01-20 18:33:55.127752: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291589 140191300101952 af.py:224] 2022-01-20 18:33:55.127756: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.291630 140191300101952 af.py:224] 2022-01-20 18:33:55.127762: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291671 140191300101952 af.py:224] 2022-01-20 18:33:55.127767: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.291712 140191300101952 af.py:224] 2022-01-20 18:33:55.127775: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.291751 140191300101952 af.py:224] 2022-01-20 18:33:55.127781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291791 140191300101952 af.py:224] 2022-01-20 18:33:55.127787: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291832 140191300101952 af.py:224] 2022-01-20 18:33:55.127792: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.291877 140191300101952 af.py:224] 2022-01-20 18:33:55.127797: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.291918 140191300101952 af.py:224] 2022-01-20 18:33:55.127802: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.291960 140191300101952 af.py:224] 2022-01-20 18:33:55.127810: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.292001 140191300101952 af.py:224] 2022-01-20 18:33:55.127818: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292042 140191300101952 af.py:224] 2022-01-20 18:33:55.127824: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292083 140191300101952 af.py:224] 2022-01-20 18:33:55.127828: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.292126 140191300101952 af.py:224] 2022-01-20 18:33:55.127835: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292168 140191300101952 af.py:224] 2022-01-20 18:33:55.127840: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.292209 140191300101952 af.py:224] 2022-01-20 18:33:55.127848: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.292249 140191300101952 af.py:224] 2022-01-20 18:33:55.127855: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292290 140191300101952 af.py:224] 2022-01-20 18:33:55.127861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292331 140191300101952 af.py:224] 2022-01-20 18:33:55.127866: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.292371 140191300101952 af.py:224] 2022-01-20 18:33:55.127872: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292413 140191300101952 af.py:224] 2022-01-20 18:33:55.127878: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.292454 140191300101952 af.py:224] 2022-01-20 18:33:55.127885: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.292494 140191300101952 af.py:224] 2022-01-20 18:33:55.127892: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292534 140191300101952 af.py:224] 2022-01-20 18:33:55.127899: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292578 140191300101952 af.py:224] 2022-01-20 18:33:55.127904: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.292619 140191300101952 af.py:224] 2022-01-20 18:33:55.127911: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292660 140191300101952 af.py:224] 2022-01-20 18:33:55.127916: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.292701 140191300101952 af.py:224] 2022-01-20 18:33:55.127924: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.292742 140191300101952 af.py:224] 2022-01-20 18:33:55.127932: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292795 140191300101952 af.py:224] 2022-01-20 18:33:55.127938: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292837 140191300101952 af.py:224] 2022-01-20 18:33:55.127944: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.292877 140191300101952 af.py:224] 2022-01-20 18:33:55.127950: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.292917 140191300101952 af.py:224] 2022-01-20 18:33:55.127955: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.292958 140191300101952 af.py:224] 2022-01-20 18:33:55.127964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.292999 140191300101952 af.py:224] 2022-01-20 18:33:55.127971: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293040 140191300101952 af.py:224] 2022-01-20 18:33:55.127978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293081 140191300101952 af.py:224] 2022-01-20 18:33:55.127983: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.293121 140191300101952 af.py:224] 2022-01-20 18:33:55.127989: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293161 140191300101952 af.py:224] 2022-01-20 18:33:55.127994: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.293202 140191300101952 af.py:224] 2022-01-20 18:33:55.128003: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.293243 140191300101952 af.py:224] 2022-01-20 18:33:55.128011: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293286 140191300101952 af.py:224] 2022-01-20 18:33:55.128017: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293327 140191300101952 af.py:224] 2022-01-20 18:33:55.128022: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.293368 140191300101952 af.py:224] 2022-01-20 18:33:55.128029: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293408 140191300101952 af.py:224] 2022-01-20 18:33:55.128034: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.293448 140191300101952 af.py:224] 2022-01-20 18:33:55.128042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.293489 140191300101952 af.py:224] 2022-01-20 18:33:55.128049: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293530 140191300101952 af.py:224] 2022-01-20 18:33:55.128055: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293570 140191300101952 af.py:224] 2022-01-20 18:33:55.128060: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.293610 140191300101952 af.py:224] 2022-01-20 18:33:55.128067: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293650 140191300101952 af.py:224] 2022-01-20 18:33:55.128071: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.293691 140191300101952 af.py:224] 2022-01-20 18:33:55.128080: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.293731 140191300101952 af.py:224] 2022-01-20 18:33:55.128087: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293772 140191300101952 af.py:224] 2022-01-20 18:33:55.128093: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293812 140191300101952 af.py:224] 2022-01-20 18:33:55.128098: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.293853 140191300101952 af.py:224] 2022-01-20 18:33:55.128105: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.293893 140191300101952 af.py:224] 2022-01-20 18:33:55.128110: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.293934 140191300101952 af.py:224] 2022-01-20 18:33:55.128123: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.293977 140191300101952 af.py:224] 2022-01-20 18:33:55.128132: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294017 140191300101952 af.py:224] 2022-01-20 18:33:55.128139: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294058 140191300101952 af.py:224] 2022-01-20 18:33:55.128144: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.294098 140191300101952 af.py:224] 2022-01-20 18:33:55.128151: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294139 140191300101952 af.py:224] 2022-01-20 18:33:55.128156: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.294180 140191300101952 af.py:224] 2022-01-20 18:33:55.128165: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.294220 140191300101952 af.py:224] 2022-01-20 18:33:55.128171: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294261 140191300101952 af.py:224] 2022-01-20 18:33:55.128177: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294303 140191300101952 af.py:224] 2022-01-20 18:33:55.128183: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.294343 140191300101952 af.py:224] 2022-01-20 18:33:55.128189: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294400 140191300101952 af.py:224] 2022-01-20 18:33:55.128195: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.294442 140191300101952 af.py:224] 2022-01-20 18:33:55.128202: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.294483 140191300101952 af.py:224] 2022-01-20 18:33:55.128209: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294542 140191300101952 af.py:224] 2022-01-20 18:33:55.128215: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294584 140191300101952 af.py:224] 2022-01-20 18:33:55.128220: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.294626 140191300101952 af.py:224] 2022-01-20 18:33:55.128227: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294685 140191300101952 af.py:224] 2022-01-20 18:33:55.128232: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.294726 140191300101952 af.py:224] 2022-01-20 18:33:55.128240: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.294766 140191300101952 af.py:224] 2022-01-20 18:33:55.128246: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294806 140191300101952 af.py:224] 2022-01-20 18:33:55.128253: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294847 140191300101952 af.py:224] 2022-01-20 18:33:55.128258: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.294888 140191300101952 af.py:224] 2022-01-20 18:33:55.128280: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.294928 140191300101952 af.py:224] 2022-01-20 18:33:55.128285: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.294969 140191300101952 af.py:224] 2022-01-20 18:33:55.128294: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.295009 140191300101952 af.py:224] 2022-01-20 18:33:55.128477: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295050 140191300101952 af.py:224] 2022-01-20 18:33:55.128485: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295091 140191300101952 af.py:224] 2022-01-20 18:33:55.128491: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.295131 140191300101952 af.py:224] 2022-01-20 18:33:55.128497: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295172 140191300101952 af.py:224] 2022-01-20 18:33:55.128502: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.295212 140191300101952 af.py:224] 2022-01-20 18:33:55.128511: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.295253 140191300101952 af.py:224] 2022-01-20 18:33:55.128519: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295294 140191300101952 af.py:224] 2022-01-20 18:33:55.128526: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295335 140191300101952 af.py:224] 2022-01-20 18:33:55.128531: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.295378 140191300101952 af.py:224] 2022-01-20 18:33:55.128538: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295420 140191300101952 af.py:224] 2022-01-20 18:33:55.128543: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.295460 140191300101952 af.py:224] 2022-01-20 18:33:55.128551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.295501 140191300101952 af.py:224] 2022-01-20 18:33:55.128558: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295542 140191300101952 af.py:224] 2022-01-20 18:33:55.128565: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295582 140191300101952 af.py:224] 2022-01-20 18:33:55.128570: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.295622 140191300101952 af.py:224] 2022-01-20 18:33:55.128576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295663 140191300101952 af.py:224] 2022-01-20 18:33:55.128582: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.295705 140191300101952 af.py:224] 2022-01-20 18:33:55.128590: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.295745 140191300101952 af.py:224] 2022-01-20 18:33:55.128597: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295785 140191300101952 af.py:224] 2022-01-20 18:33:55.128603: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295826 140191300101952 af.py:224] 2022-01-20 18:33:55.128608: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.295867 140191300101952 af.py:224] 2022-01-20 18:33:55.128615: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.295908 140191300101952 af.py:224] 2022-01-20 18:33:55.128620: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.295948 140191300101952 af.py:224] 2022-01-20 18:33:55.128628: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.295988 140191300101952 af.py:224] 2022-01-20 18:33:55.128635: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296042 140191300101952 af.py:224] 2022-01-20 18:33:55.128642: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296087 140191300101952 af.py:224] 2022-01-20 18:33:55.128647: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.296132 140191300101952 af.py:224] 2022-01-20 18:33:55.128653: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296174 140191300101952 af.py:224] 2022-01-20 18:33:55.128658: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.296215 140191300101952 af.py:224] 2022-01-20 18:33:55.128667: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.296256 140191300101952 af.py:224] 2022-01-20 18:33:55.128673: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296298 140191300101952 af.py:224] 2022-01-20 18:33:55.128680: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296339 140191300101952 af.py:224] 2022-01-20 18:33:55.128685: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.296380 140191300101952 af.py:224] 2022-01-20 18:33:55.128690: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296421 140191300101952 af.py:224] 2022-01-20 18:33:55.128695: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.296462 140191300101952 af.py:224] 2022-01-20 18:33:55.128703: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.296503 140191300101952 af.py:224] 2022-01-20 18:33:55.128710: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296543 140191300101952 af.py:224] 2022-01-20 18:33:55.128716: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296584 140191300101952 af.py:224] 2022-01-20 18:33:55.128721: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.296625 140191300101952 af.py:224] 2022-01-20 18:33:55.128726: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296666 140191300101952 af.py:224] 2022-01-20 18:33:55.128731: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.296707 140191300101952 af.py:224] 2022-01-20 18:33:55.128739: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.296747 140191300101952 af.py:224] 2022-01-20 18:33:55.128745: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296791 140191300101952 af.py:224] 2022-01-20 18:33:55.128751: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296833 140191300101952 af.py:224] 2022-01-20 18:33:55.128756: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.296874 140191300101952 af.py:224] 2022-01-20 18:33:55.128762: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.296915 140191300101952 af.py:224] 2022-01-20 18:33:55.128767: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.296955 140191300101952 af.py:224] 2022-01-20 18:33:55.128774: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.296996 140191300101952 af.py:224] 2022-01-20 18:33:55.128781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297037 140191300101952 af.py:224] 2022-01-20 18:33:55.128787: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297078 140191300101952 af.py:224] 2022-01-20 18:33:55.128791: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.297118 140191300101952 af.py:224] 2022-01-20 18:33:55.128797: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297159 140191300101952 af.py:224] 2022-01-20 18:33:55.128802: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.297200 140191300101952 af.py:224] 2022-01-20 18:33:55.128810: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.297240 140191300101952 af.py:224] 2022-01-20 18:33:55.128817: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297280 140191300101952 af.py:224] 2022-01-20 18:33:55.128823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297321 140191300101952 af.py:224] 2022-01-20 18:33:55.128828: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.297362 140191300101952 af.py:224] 2022-01-20 18:33:55.128835: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297403 140191300101952 af.py:224] 2022-01-20 18:33:55.128840: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.297444 140191300101952 af.py:224] 2022-01-20 18:33:55.128848: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.297487 140191300101952 af.py:224] 2022-01-20 18:33:55.128854: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297528 140191300101952 af.py:224] 2022-01-20 18:33:55.128860: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297569 140191300101952 af.py:224] 2022-01-20 18:33:55.128864: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.297610 140191300101952 af.py:224] 2022-01-20 18:33:55.128870: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297665 140191300101952 af.py:224] 2022-01-20 18:33:55.128875: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.297707 140191300101952 af.py:224] 2022-01-20 18:33:55.128883: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.297747 140191300101952 af.py:224] 2022-01-20 18:33:55.128890: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297788 140191300101952 af.py:224] 2022-01-20 18:33:55.128897: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297829 140191300101952 af.py:224] 2022-01-20 18:33:55.128901: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.297870 140191300101952 af.py:224] 2022-01-20 18:33:55.128907: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.297910 140191300101952 af.py:224] 2022-01-20 18:33:55.128912: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.297950 140191300101952 af.py:224] 2022-01-20 18:33:55.128920: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.297991 140191300101952 af.py:224] 2022-01-20 18:33:55.128927: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298031 140191300101952 af.py:224] 2022-01-20 18:33:55.128934: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298071 140191300101952 af.py:224] 2022-01-20 18:33:55.128938: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.298112 140191300101952 af.py:224] 2022-01-20 18:33:55.128945: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298156 140191300101952 af.py:224] 2022-01-20 18:33:55.128950: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.298198 140191300101952 af.py:224] 2022-01-20 18:33:55.128958: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.298238 140191300101952 af.py:224] 2022-01-20 18:33:55.128964: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298278 140191300101952 af.py:224] 2022-01-20 18:33:55.128970: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298319 140191300101952 af.py:224] 2022-01-20 18:33:55.128975: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.298360 140191300101952 af.py:224] 2022-01-20 18:33:55.128980: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298401 140191300101952 af.py:224] 2022-01-20 18:33:55.128985: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.298441 140191300101952 af.py:224] 2022-01-20 18:33:55.128993: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.298482 140191300101952 af.py:224] 2022-01-20 18:33:55.129000: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298523 140191300101952 af.py:224] 2022-01-20 18:33:55.129005: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298563 140191300101952 af.py:224] 2022-01-20 18:33:55.129010: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.298604 140191300101952 af.py:224] 2022-01-20 18:33:55.129016: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298645 140191300101952 af.py:224] 2022-01-20 18:33:55.129021: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.298685 140191300101952 af.py:224] 2022-01-20 18:33:55.129029: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.298725 140191300101952 af.py:224] 2022-01-20 18:33:55.129035: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298766 140191300101952 af.py:224] 2022-01-20 18:33:55.129042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298806 140191300101952 af.py:224] 2022-01-20 18:33:55.129047: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.298850 140191300101952 af.py:224] 2022-01-20 18:33:55.129052: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.298891 140191300101952 af.py:224] 2022-01-20 18:33:55.129057: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.298932 140191300101952 af.py:224] 2022-01-20 18:33:55.129066: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.298972 140191300101952 af.py:224] 2022-01-20 18:33:55.129073: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299013 140191300101952 af.py:224] 2022-01-20 18:33:55.129079: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299053 140191300101952 af.py:224] 2022-01-20 18:33:55.129084: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.299094 140191300101952 af.py:224] 2022-01-20 18:33:55.129090: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299135 140191300101952 af.py:224] 2022-01-20 18:33:55.129096: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.299175 140191300101952 af.py:224] 2022-01-20 18:33:55.129104: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.299216 140191300101952 af.py:224] 2022-01-20 18:33:55.129111: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299271 140191300101952 af.py:224] 2022-01-20 18:33:55.129117: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299314 140191300101952 af.py:224] 2022-01-20 18:33:55.129122: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.299354 140191300101952 af.py:224] 2022-01-20 18:33:55.129128: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299395 140191300101952 af.py:224] 2022-01-20 18:33:55.129133: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.299436 140191300101952 af.py:224] 2022-01-20 18:33:55.129141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.299476 140191300101952 af.py:224] 2022-01-20 18:33:55.129148: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299516 140191300101952 af.py:224] 2022-01-20 18:33:55.129153: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299561 140191300101952 af.py:224] 2022-01-20 18:33:55.129158: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.299602 140191300101952 af.py:224] 2022-01-20 18:33:55.129164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299642 140191300101952 af.py:224] 2022-01-20 18:33:55.129170: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.299682 140191300101952 af.py:224] 2022-01-20 18:33:55.129178: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.299723 140191300101952 af.py:224] 2022-01-20 18:33:55.129185: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299764 140191300101952 af.py:224] 2022-01-20 18:33:55.129191: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299804 140191300101952 af.py:224] 2022-01-20 18:33:55.129196: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.299845 140191300101952 af.py:224] 2022-01-20 18:33:55.129203: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.299885 140191300101952 af.py:224] 2022-01-20 18:33:55.129208: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.299926 140191300101952 af.py:224] 2022-01-20 18:33:55.129216: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.299965 140191300101952 af.py:224] 2022-01-20 18:33:55.129223: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300006 140191300101952 af.py:224] 2022-01-20 18:33:55.129229: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300047 140191300101952 af.py:224] 2022-01-20 18:33:55.129233: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.300088 140191300101952 af.py:224] 2022-01-20 18:33:55.129239: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300168 140191300101952 af.py:224] 2022-01-20 18:33:55.129244: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.300209 140191300101952 af.py:224] 2022-01-20 18:33:55.129252: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.300250 140191300101952 af.py:224] 2022-01-20 18:33:55.129259: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300293 140191300101952 af.py:224] 2022-01-20 18:33:55.129266: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300334 140191300101952 af.py:224] 2022-01-20 18:33:55.129271: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.300375 140191300101952 af.py:224] 2022-01-20 18:33:55.129277: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300416 140191300101952 af.py:224] 2022-01-20 18:33:55.129282: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.300457 140191300101952 af.py:224] 2022-01-20 18:33:55.129290: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.300498 140191300101952 af.py:224] 2022-01-20 18:33:55.129296: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300538 140191300101952 af.py:224] 2022-01-20 18:33:55.129303: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300578 140191300101952 af.py:224] 2022-01-20 18:33:55.129307: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.300619 140191300101952 af.py:224] 2022-01-20 18:33:55.129313: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300659 140191300101952 af.py:224] 2022-01-20 18:33:55.129318: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.300700 140191300101952 af.py:224] 2022-01-20 18:33:55.129326: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.300740 140191300101952 af.py:224] 2022-01-20 18:33:55.129332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300781 140191300101952 af.py:224] 2022-01-20 18:33:55.129339: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300822 140191300101952 af.py:224] 2022-01-20 18:33:55.129344: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.300863 140191300101952 af.py:224] 2022-01-20 18:33:55.129350: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.300904 140191300101952 af.py:224] 2022-01-20 18:33:55.129355: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.300959 140191300101952 af.py:224] 2022-01-20 18:33:55.129364: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.301005 140191300101952 af.py:224] 2022-01-20 18:33:55.129370: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301047 140191300101952 af.py:224] 2022-01-20 18:33:55.129376: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301088 140191300101952 af.py:224] 2022-01-20 18:33:55.129381: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.301161 140191300101952 af.py:224] 2022-01-20 18:33:55.129387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301234 140191300101952 af.py:224] 2022-01-20 18:33:55.129392: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.301274 140191300101952 af.py:224] 2022-01-20 18:33:55.129400: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.301314 140191300101952 af.py:224] 2022-01-20 18:33:55.129407: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301355 140191300101952 af.py:224] 2022-01-20 18:33:55.129414: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301396 140191300101952 af.py:224] 2022-01-20 18:33:55.129419: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.301436 140191300101952 af.py:224] 2022-01-20 18:33:55.129424: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301477 140191300101952 af.py:224] 2022-01-20 18:33:55.129429: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.301517 140191300101952 af.py:224] 2022-01-20 18:33:55.129438: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.301558 140191300101952 af.py:224] 2022-01-20 18:33:55.129445: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301599 140191300101952 af.py:224] 2022-01-20 18:33:55.129451: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301640 140191300101952 af.py:224] 2022-01-20 18:33:55.129456: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.301680 140191300101952 af.py:224] 2022-01-20 18:33:55.129463: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301724 140191300101952 af.py:224] 2022-01-20 18:33:55.129468: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.301765 140191300101952 af.py:224] 2022-01-20 18:33:55.129476: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.301806 140191300101952 af.py:224] 2022-01-20 18:33:55.129483: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301846 140191300101952 af.py:224] 2022-01-20 18:33:55.129489: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301887 140191300101952 af.py:224] 2022-01-20 18:33:55.129494: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.301928 140191300101952 af.py:224] 2022-01-20 18:33:55.129501: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.301969 140191300101952 af.py:224] 2022-01-20 18:33:55.129506: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.302009 140191300101952 af.py:224] 2022-01-20 18:33:55.129514: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.302050 140191300101952 af.py:224] 2022-01-20 18:33:55.129521: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302091 140191300101952 af.py:224] 2022-01-20 18:33:55.129527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302131 140191300101952 af.py:224] 2022-01-20 18:33:55.129532: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.302172 140191300101952 af.py:224] 2022-01-20 18:33:55.129539: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302212 140191300101952 af.py:224] 2022-01-20 18:33:55.129544: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.302253 140191300101952 af.py:224] 2022-01-20 18:33:55.129552: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.302294 140191300101952 af.py:224] 2022-01-20 18:33:55.129559: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302335 140191300101952 af.py:224] 2022-01-20 18:33:55.129565: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302376 140191300101952 af.py:224] 2022-01-20 18:33:55.129570: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.302422 140191300101952 af.py:224] 2022-01-20 18:33:55.129576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302464 140191300101952 af.py:224] 2022-01-20 18:33:55.129581: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.302504 140191300101952 af.py:224] 2022-01-20 18:33:55.129590: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.302544 140191300101952 af.py:224] 2022-01-20 18:33:55.129596: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302585 140191300101952 af.py:224] 2022-01-20 18:33:55.129603: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302639 140191300101952 af.py:224] 2022-01-20 18:33:55.129607: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.302680 140191300101952 af.py:224] 2022-01-20 18:33:55.129614: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302721 140191300101952 af.py:224] 2022-01-20 18:33:55.129619: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.302761 140191300101952 af.py:224] 2022-01-20 18:33:55.129627: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.302801 140191300101952 af.py:224] 2022-01-20 18:33:55.129635: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302842 140191300101952 af.py:224] 2022-01-20 18:33:55.129642: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302882 140191300101952 af.py:224] 2022-01-20 18:33:55.129646: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.302923 140191300101952 af.py:224] 2022-01-20 18:33:55.129652: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.302963 140191300101952 af.py:224] 2022-01-20 18:33:55.129658: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.303003 140191300101952 af.py:224] 2022-01-20 18:33:55.129666: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.303044 140191300101952 af.py:224] 2022-01-20 18:33:55.129673: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303085 140191300101952 af.py:224] 2022-01-20 18:33:55.129679: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303129 140191300101952 af.py:224] 2022-01-20 18:33:55.129684: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.303169 140191300101952 af.py:224] 2022-01-20 18:33:55.129691: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303210 140191300101952 af.py:224] 2022-01-20 18:33:55.129696: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.303250 140191300101952 af.py:224] 2022-01-20 18:33:55.129704: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.303290 140191300101952 af.py:224] 2022-01-20 18:33:55.129711: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303331 140191300101952 af.py:224] 2022-01-20 18:33:55.129717: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303371 140191300101952 af.py:224] 2022-01-20 18:33:55.129722: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.303411 140191300101952 af.py:224] 2022-01-20 18:33:55.129729: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303452 140191300101952 af.py:224] 2022-01-20 18:33:55.129734: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.303492 140191300101952 af.py:224] 2022-01-20 18:33:55.129743: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.303533 140191300101952 af.py:224] 2022-01-20 18:33:55.129750: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303574 140191300101952 af.py:224] 2022-01-20 18:33:55.129756: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303614 140191300101952 af.py:224] 2022-01-20 18:33:55.129761: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.303654 140191300101952 af.py:224] 2022-01-20 18:33:55.129768: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303695 140191300101952 af.py:224] 2022-01-20 18:33:55.129773: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.303735 140191300101952 af.py:224] 2022-01-20 18:33:55.129781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.303775 140191300101952 af.py:224] 2022-01-20 18:33:55.129788: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303819 140191300101952 af.py:224] 2022-01-20 18:33:55.129794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303860 140191300101952 af.py:224] 2022-01-20 18:33:55.129799: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.303900 140191300101952 af.py:224] 2022-01-20 18:33:55.129806: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.303941 140191300101952 af.py:224] 2022-01-20 18:33:55.129811: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.303982 140191300101952 af.py:224] 2022-01-20 18:33:55.129819: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.304022 140191300101952 af.py:224] 2022-01-20 18:33:55.129826: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304062 140191300101952 af.py:224] 2022-01-20 18:33:55.129832: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304102 140191300101952 af.py:224] 2022-01-20 18:33:55.129837: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.304147 140191300101952 af.py:224] 2022-01-20 18:33:55.129844: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304188 140191300101952 af.py:224] 2022-01-20 18:33:55.129849: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.304242 140191300101952 af.py:224] 2022-01-20 18:33:55.129857: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.304284 140191300101952 af.py:224] 2022-01-20 18:33:55.129864: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304324 140191300101952 af.py:224] 2022-01-20 18:33:55.129871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304365 140191300101952 af.py:224] 2022-01-20 18:33:55.129876: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.304406 140191300101952 af.py:224] 2022-01-20 18:33:55.129882: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304446 140191300101952 af.py:224] 2022-01-20 18:33:55.129887: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.304487 140191300101952 af.py:224] 2022-01-20 18:33:55.129896: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.304530 140191300101952 af.py:224] 2022-01-20 18:33:55.129902: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304571 140191300101952 af.py:224] 2022-01-20 18:33:55.129908: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304612 140191300101952 af.py:224] 2022-01-20 18:33:55.129914: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.304653 140191300101952 af.py:224] 2022-01-20 18:33:55.129921: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304693 140191300101952 af.py:224] 2022-01-20 18:33:55.129926: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.304733 140191300101952 af.py:224] 2022-01-20 18:33:55.129934: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.304774 140191300101952 af.py:224] 2022-01-20 18:33:55.129941: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304815 140191300101952 af.py:224] 2022-01-20 18:33:55.129947: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304855 140191300101952 af.py:224] 2022-01-20 18:33:55.129952: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.304896 140191300101952 af.py:224] 2022-01-20 18:33:55.129959: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.304936 140191300101952 af.py:224] 2022-01-20 18:33:55.129964: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.304976 140191300101952 af.py:224] 2022-01-20 18:33:55.129972: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.305016 140191300101952 af.py:224] 2022-01-20 18:33:55.129978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305057 140191300101952 af.py:224] 2022-01-20 18:33:55.129984: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305098 140191300101952 af.py:224] 2022-01-20 18:33:55.129990: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.305138 140191300101952 af.py:224] 2022-01-20 18:33:55.129995: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305182 140191300101952 af.py:224] 2022-01-20 18:33:55.130000: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.305223 140191300101952 af.py:224] 2022-01-20 18:33:55.130008: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.305264 140191300101952 af.py:224] 2022-01-20 18:33:55.130016: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305305 140191300101952 af.py:224] 2022-01-20 18:33:55.130023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305346 140191300101952 af.py:224] 2022-01-20 18:33:55.130028: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.305387 140191300101952 af.py:224] 2022-01-20 18:33:55.130034: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305427 140191300101952 af.py:224] 2022-01-20 18:33:55.130039: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.305467 140191300101952 af.py:224] 2022-01-20 18:33:55.130047: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.305508 140191300101952 af.py:224] 2022-01-20 18:33:55.130053: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305549 140191300101952 af.py:224] 2022-01-20 18:33:55.130058: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305589 140191300101952 af.py:224] 2022-01-20 18:33:55.130063: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.305629 140191300101952 af.py:224] 2022-01-20 18:33:55.130069: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305670 140191300101952 af.py:224] 2022-01-20 18:33:55.130074: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.305711 140191300101952 af.py:224] 2022-01-20 18:33:55.130082: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.305752 140191300101952 af.py:224] 2022-01-20 18:33:55.130089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305792 140191300101952 af.py:224] 2022-01-20 18:33:55.130095: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305846 140191300101952 af.py:224] 2022-01-20 18:33:55.130099: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.305891 140191300101952 af.py:224] 2022-01-20 18:33:55.130105: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.305933 140191300101952 af.py:224] 2022-01-20 18:33:55.130111: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.305974 140191300101952 af.py:224] 2022-01-20 18:33:55.130119: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.306014 140191300101952 af.py:224] 2022-01-20 18:33:55.130127: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306055 140191300101952 af.py:224] 2022-01-20 18:33:55.130133: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306095 140191300101952 af.py:224] 2022-01-20 18:33:55.130138: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.306136 140191300101952 af.py:224] 2022-01-20 18:33:55.130145: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306177 140191300101952 af.py:224] 2022-01-20 18:33:55.130150: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.306218 140191300101952 af.py:224] 2022-01-20 18:33:55.130158: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.306258 140191300101952 af.py:224] 2022-01-20 18:33:55.130165: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306299 140191300101952 af.py:224] 2022-01-20 18:33:55.130171: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306339 140191300101952 af.py:224] 2022-01-20 18:33:55.130176: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.306379 140191300101952 af.py:224] 2022-01-20 18:33:55.130182: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306419 140191300101952 af.py:224] 2022-01-20 18:33:55.130187: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.306460 140191300101952 af.py:224] 2022-01-20 18:33:55.130194: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.306501 140191300101952 af.py:224] 2022-01-20 18:33:55.130201: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306541 140191300101952 af.py:224] 2022-01-20 18:33:55.130207: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306585 140191300101952 af.py:224] 2022-01-20 18:33:55.130212: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.306626 140191300101952 af.py:224] 2022-01-20 18:33:55.130219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306667 140191300101952 af.py:224] 2022-01-20 18:33:55.130224: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.306707 140191300101952 af.py:224] 2022-01-20 18:33:55.130232: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.306748 140191300101952 af.py:224] 2022-01-20 18:33:55.130266: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306788 140191300101952 af.py:224] 2022-01-20 18:33:55.130274: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306828 140191300101952 af.py:224] 2022-01-20 18:33:55.130279: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.306869 140191300101952 af.py:224] 2022-01-20 18:33:55.130285: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.306910 140191300101952 af.py:224] 2022-01-20 18:33:55.130290: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.306951 140191300101952 af.py:224] 2022-01-20 18:33:55.130297: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.306991 140191300101952 af.py:224] 2022-01-20 18:33:55.130304: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307033 140191300101952 af.py:224] 2022-01-20 18:33:55.130310: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307073 140191300101952 af.py:224] 2022-01-20 18:33:55.130314: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.307114 140191300101952 af.py:224] 2022-01-20 18:33:55.130320: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307155 140191300101952 af.py:224] 2022-01-20 18:33:55.130325: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.307196 140191300101952 af.py:224] 2022-01-20 18:33:55.130333: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.307237 140191300101952 af.py:224] 2022-01-20 18:33:55.130339: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307281 140191300101952 af.py:224] 2022-01-20 18:33:55.130345: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307322 140191300101952 af.py:224] 2022-01-20 18:33:55.130350: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.307362 140191300101952 af.py:224] 2022-01-20 18:33:55.130357: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307403 140191300101952 af.py:224] 2022-01-20 18:33:55.130362: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.307444 140191300101952 af.py:224] 2022-01-20 18:33:55.130369: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.307501 140191300101952 af.py:224] 2022-01-20 18:33:55.130376: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307544 140191300101952 af.py:224] 2022-01-20 18:33:55.130382: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307585 140191300101952 af.py:224] 2022-01-20 18:33:55.130387: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.307657 140191300101952 af.py:224] 2022-01-20 18:33:55.130393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307698 140191300101952 af.py:224] 2022-01-20 18:33:55.130398: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.307738 140191300101952 af.py:224] 2022-01-20 18:33:55.130406: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.307779 140191300101952 af.py:224] 2022-01-20 18:33:55.130414: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307819 140191300101952 af.py:224] 2022-01-20 18:33:55.130420: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307860 140191300101952 af.py:224] 2022-01-20 18:33:55.130424: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.307900 140191300101952 af.py:224] 2022-01-20 18:33:55.130430: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.307941 140191300101952 af.py:224] 2022-01-20 18:33:55.130435: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.307985 140191300101952 af.py:224] 2022-01-20 18:33:55.130443: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.308025 140191300101952 af.py:224] 2022-01-20 18:33:55.130449: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308067 140191300101952 af.py:224] 2022-01-20 18:33:55.130455: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308107 140191300101952 af.py:224] 2022-01-20 18:33:55.130460: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.308151 140191300101952 af.py:224] 2022-01-20 18:33:55.130466: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308192 140191300101952 af.py:224] 2022-01-20 18:33:55.130471: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.308233 140191300101952 af.py:224] 2022-01-20 18:33:55.130479: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.308273 140191300101952 af.py:224] 2022-01-20 18:33:55.130485: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308313 140191300101952 af.py:224] 2022-01-20 18:33:55.130492: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308354 140191300101952 af.py:224] 2022-01-20 18:33:55.130497: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.308395 140191300101952 af.py:224] 2022-01-20 18:33:55.130505: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308436 140191300101952 af.py:224] 2022-01-20 18:33:55.130510: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.308476 140191300101952 af.py:224] 2022-01-20 18:33:55.130519: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.308516 140191300101952 af.py:224] 2022-01-20 18:33:55.130526: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308557 140191300101952 af.py:224] 2022-01-20 18:33:55.130532: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308598 140191300101952 af.py:224] 2022-01-20 18:33:55.130537: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.308638 140191300101952 af.py:224] 2022-01-20 18:33:55.130542: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308682 140191300101952 af.py:224] 2022-01-20 18:33:55.130548: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.308723 140191300101952 af.py:224] 2022-01-20 18:33:55.130556: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.308764 140191300101952 af.py:224] 2022-01-20 18:33:55.130563: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308805 140191300101952 af.py:224] 2022-01-20 18:33:55.130569: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308846 140191300101952 af.py:224] 2022-01-20 18:33:55.130575: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.308887 140191300101952 af.py:224] 2022-01-20 18:33:55.130581: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.308928 140191300101952 af.py:224] 2022-01-20 18:33:55.130586: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.308969 140191300101952 af.py:224] 2022-01-20 18:33:55.130595: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.309010 140191300101952 af.py:224] 2022-01-20 18:33:55.130601: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309050 140191300101952 af.py:224] 2022-01-20 18:33:55.130607: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309091 140191300101952 af.py:224] 2022-01-20 18:33:55.130612: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.309145 140191300101952 af.py:224] 2022-01-20 18:33:55.130618: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309186 140191300101952 af.py:224] 2022-01-20 18:33:55.130623: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.309226 140191300101952 af.py:224] 2022-01-20 18:33:55.130632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.309267 140191300101952 af.py:224] 2022-01-20 18:33:55.130639: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309307 140191300101952 af.py:224] 2022-01-20 18:33:55.130645: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309348 140191300101952 af.py:224] 2022-01-20 18:33:55.130651: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.309392 140191300101952 af.py:224] 2022-01-20 18:33:55.130657: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309433 140191300101952 af.py:224] 2022-01-20 18:33:55.130662: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.309474 140191300101952 af.py:224] 2022-01-20 18:33:55.130671: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.309514 140191300101952 af.py:224] 2022-01-20 18:33:55.130677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309554 140191300101952 af.py:224] 2022-01-20 18:33:55.130684: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309594 140191300101952 af.py:224] 2022-01-20 18:33:55.130689: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.309635 140191300101952 af.py:224] 2022-01-20 18:33:55.130695: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309674 140191300101952 af.py:224] 2022-01-20 18:33:55.130700: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.309715 140191300101952 af.py:224] 2022-01-20 18:33:55.130709: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.309755 140191300101952 af.py:224] 2022-01-20 18:33:55.130717: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309795 140191300101952 af.py:224] 2022-01-20 18:33:55.130723: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309836 140191300101952 af.py:224] 2022-01-20 18:33:55.130728: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.309876 140191300101952 af.py:224] 2022-01-20 18:33:55.130735: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.309917 140191300101952 af.py:224] 2022-01-20 18:33:55.130740: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.309958 140191300101952 af.py:224] 2022-01-20 18:33:55.130748: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.309999 140191300101952 af.py:224] 2022-01-20 18:33:55.130755: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310039 140191300101952 af.py:224] 2022-01-20 18:33:55.130762: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310088 140191300101952 af.py:224] 2022-01-20 18:33:55.130767: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.310129 140191300101952 af.py:224] 2022-01-20 18:33:55.130773: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310170 140191300101952 af.py:224] 2022-01-20 18:33:55.130778: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.310210 140191300101952 af.py:224] 2022-01-20 18:33:55.130786: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.310251 140191300101952 af.py:224] 2022-01-20 18:33:55.130793: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310292 140191300101952 af.py:224] 2022-01-20 18:33:55.130800: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310332 140191300101952 af.py:224] 2022-01-20 18:33:55.130805: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.310373 140191300101952 af.py:224] 2022-01-20 18:33:55.130811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310413 140191300101952 af.py:224] 2022-01-20 18:33:55.130816: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.310454 140191300101952 af.py:224] 2022-01-20 18:33:55.130825: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.310494 140191300101952 af.py:224] 2022-01-20 18:33:55.130832: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310536 140191300101952 af.py:224] 2022-01-20 18:33:55.130838: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310577 140191300101952 af.py:224] 2022-01-20 18:33:55.130843: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.310618 140191300101952 af.py:224] 2022-01-20 18:33:55.130851: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310658 140191300101952 af.py:224] 2022-01-20 18:33:55.130856: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.310699 140191300101952 af.py:224] 2022-01-20 18:33:55.130864: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.310752 140191300101952 af.py:224] 2022-01-20 18:33:55.130871: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310797 140191300101952 af.py:224] 2022-01-20 18:33:55.130878: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310838 140191300101952 af.py:224] 2022-01-20 18:33:55.130883: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.310879 140191300101952 af.py:224] 2022-01-20 18:33:55.130889: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.310920 140191300101952 af.py:224] 2022-01-20 18:33:55.130894: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.310960 140191300101952 af.py:224] 2022-01-20 18:33:55.130903: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.311000 140191300101952 af.py:224] 2022-01-20 18:33:55.130910: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311040 140191300101952 af.py:224] 2022-01-20 18:33:55.130917: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311081 140191300101952 af.py:224] 2022-01-20 18:33:55.130922: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.311121 140191300101952 af.py:224] 2022-01-20 18:33:55.130927: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311161 140191300101952 af.py:224] 2022-01-20 18:33:55.130932: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.311201 140191300101952 af.py:224] 2022-01-20 18:33:55.130940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.311241 140191300101952 af.py:224] 2022-01-20 18:33:55.130947: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311282 140191300101952 af.py:224] 2022-01-20 18:33:55.130953: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311323 140191300101952 af.py:224] 2022-01-20 18:33:55.130959: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.311364 140191300101952 af.py:224] 2022-01-20 18:33:55.130965: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311405 140191300101952 af.py:224] 2022-01-20 18:33:55.130970: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.311449 140191300101952 af.py:224] 2022-01-20 18:33:55.130978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.311490 140191300101952 af.py:224] 2022-01-20 18:33:55.130985: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311531 140191300101952 af.py:224] 2022-01-20 18:33:55.130991: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311571 140191300101952 af.py:224] 2022-01-20 18:33:55.130996: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.311612 140191300101952 af.py:224] 2022-01-20 18:33:55.131003: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311652 140191300101952 af.py:224] 2022-01-20 18:33:55.131009: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.311693 140191300101952 af.py:224] 2022-01-20 18:33:55.131017: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.311734 140191300101952 af.py:224] 2022-01-20 18:33:55.131023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311774 140191300101952 af.py:224] 2022-01-20 18:33:55.131030: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311814 140191300101952 af.py:224] 2022-01-20 18:33:55.131035: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.311855 140191300101952 af.py:224] 2022-01-20 18:33:55.131041: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.311896 140191300101952 af.py:224] 2022-01-20 18:33:55.131046: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.311936 140191300101952 af.py:224] 2022-01-20 18:33:55.131054: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.311976 140191300101952 af.py:224] 2022-01-20 18:33:55.131061: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312016 140191300101952 af.py:224] 2022-01-20 18:33:55.131067: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312057 140191300101952 af.py:224] 2022-01-20 18:33:55.131072: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.312097 140191300101952 af.py:224] 2022-01-20 18:33:55.131079: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312145 140191300101952 af.py:224] 2022-01-20 18:33:55.131084: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.312187 140191300101952 af.py:224] 2022-01-20 18:33:55.131092: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.312228 140191300101952 af.py:224] 2022-01-20 18:33:55.131099: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312269 140191300101952 af.py:224] 2022-01-20 18:33:55.131105: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312310 140191300101952 af.py:224] 2022-01-20 18:33:55.131111: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.312364 140191300101952 af.py:224] 2022-01-20 18:33:55.131117: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312406 140191300101952 af.py:224] 2022-01-20 18:33:55.131122: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.312447 140191300101952 af.py:224] 2022-01-20 18:33:55.131132: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.312488 140191300101952 af.py:224] 2022-01-20 18:33:55.131138: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312528 140191300101952 af.py:224] 2022-01-20 18:33:55.131145: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312568 140191300101952 af.py:224] 2022-01-20 18:33:55.131150: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.312609 140191300101952 af.py:224] 2022-01-20 18:33:55.131157: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312650 140191300101952 af.py:224] 2022-01-20 18:33:55.131162: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.312691 140191300101952 af.py:224] 2022-01-20 18:33:55.131170: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.312731 140191300101952 af.py:224] 2022-01-20 18:33:55.131178: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312772 140191300101952 af.py:224] 2022-01-20 18:33:55.131185: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312813 140191300101952 af.py:224] 2022-01-20 18:33:55.131191: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.312858 140191300101952 af.py:224] 2022-01-20 18:33:55.131197: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.312899 140191300101952 af.py:224] 2022-01-20 18:33:55.131202: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.312940 140191300101952 af.py:224] 2022-01-20 18:33:55.131211: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.312980 140191300101952 af.py:224] 2022-01-20 18:33:55.131218: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313021 140191300101952 af.py:224] 2022-01-20 18:33:55.131224: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313062 140191300101952 af.py:224] 2022-01-20 18:33:55.131229: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.313103 140191300101952 af.py:224] 2022-01-20 18:33:55.131236: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313143 140191300101952 af.py:224] 2022-01-20 18:33:55.131241: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.313184 140191300101952 af.py:224] 2022-01-20 18:33:55.131250: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.313225 140191300101952 af.py:224] 2022-01-20 18:33:55.131257: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313265 140191300101952 af.py:224] 2022-01-20 18:33:55.131264: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313306 140191300101952 af.py:224] 2022-01-20 18:33:55.131268: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.313346 140191300101952 af.py:224] 2022-01-20 18:33:55.131275: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313387 140191300101952 af.py:224] 2022-01-20 18:33:55.131280: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.313428 140191300101952 af.py:224] 2022-01-20 18:33:55.131289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.313469 140191300101952 af.py:224] 2022-01-20 18:33:55.131297: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313510 140191300101952 af.py:224] 2022-01-20 18:33:55.131303: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313553 140191300101952 af.py:224] 2022-01-20 18:33:55.131308: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.313594 140191300101952 af.py:224] 2022-01-20 18:33:55.131315: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313636 140191300101952 af.py:224] 2022-01-20 18:33:55.131320: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.313676 140191300101952 af.py:224] 2022-01-20 18:33:55.131329: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.313716 140191300101952 af.py:224] 2022-01-20 18:33:55.131335: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313756 140191300101952 af.py:224] 2022-01-20 18:33:55.131341: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313797 140191300101952 af.py:224] 2022-01-20 18:33:55.131346: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.313838 140191300101952 af.py:224] 2022-01-20 18:33:55.131352: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.313878 140191300101952 af.py:224] 2022-01-20 18:33:55.131357: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.313919 140191300101952 af.py:224] 2022-01-20 18:33:55.131435: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.313976 140191300101952 af.py:224] 2022-01-20 18:33:55.131445: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314018 140191300101952 af.py:224] 2022-01-20 18:33:55.131451: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314086 140191300101952 af.py:224] 2022-01-20 18:33:55.131456: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.314127 140191300101952 af.py:224] 2022-01-20 18:33:55.131463: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314182 140191300101952 af.py:224] 2022-01-20 18:33:55.131468: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.314223 140191300101952 af.py:224] 2022-01-20 18:33:55.131476: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.314282 140191300101952 af.py:224] 2022-01-20 18:33:55.131483: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314341 140191300101952 af.py:224] 2022-01-20 18:33:55.131489: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314382 140191300101952 af.py:224] 2022-01-20 18:33:55.131494: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.314423 140191300101952 af.py:224] 2022-01-20 18:33:55.131501: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314464 140191300101952 af.py:224] 2022-01-20 18:33:55.131506: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.314504 140191300101952 af.py:224] 2022-01-20 18:33:55.131514: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.314545 140191300101952 af.py:224] 2022-01-20 18:33:55.131521: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314585 140191300101952 af.py:224] 2022-01-20 18:33:55.131527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314626 140191300101952 af.py:224] 2022-01-20 18:33:55.131532: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.314666 140191300101952 af.py:224] 2022-01-20 18:33:55.131539: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314707 140191300101952 af.py:224] 2022-01-20 18:33:55.131544: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.314748 140191300101952 af.py:224] 2022-01-20 18:33:55.131551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.314789 140191300101952 af.py:224] 2022-01-20 18:33:55.131558: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314829 140191300101952 af.py:224] 2022-01-20 18:33:55.131565: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314870 140191300101952 af.py:224] 2022-01-20 18:33:55.131570: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.314910 140191300101952 af.py:224] 2022-01-20 18:33:55.131576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.314950 140191300101952 af.py:224] 2022-01-20 18:33:55.131582: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.314993 140191300101952 af.py:224] 2022-01-20 18:33:55.131590: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.315033 140191300101952 af.py:224] 2022-01-20 18:33:55.131596: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315073 140191300101952 af.py:224] 2022-01-20 18:33:55.131602: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315114 140191300101952 af.py:224] 2022-01-20 18:33:55.131607: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.315154 140191300101952 af.py:224] 2022-01-20 18:33:55.131614: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315195 140191300101952 af.py:224] 2022-01-20 18:33:55.131619: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.315235 140191300101952 af.py:224] 2022-01-20 18:33:55.131627: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.315276 140191300101952 af.py:224] 2022-01-20 18:33:55.131635: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315317 140191300101952 af.py:224] 2022-01-20 18:33:55.131641: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315358 140191300101952 af.py:224] 2022-01-20 18:33:55.131646: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.315399 140191300101952 af.py:224] 2022-01-20 18:33:55.131653: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315440 140191300101952 af.py:224] 2022-01-20 18:33:55.131658: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.315481 140191300101952 af.py:224] 2022-01-20 18:33:55.131666: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.315522 140191300101952 af.py:224] 2022-01-20 18:33:55.131673: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315563 140191300101952 af.py:224] 2022-01-20 18:33:55.131679: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315604 140191300101952 af.py:224] 2022-01-20 18:33:55.131685: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.315660 140191300101952 af.py:224] 2022-01-20 18:33:55.131691: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315705 140191300101952 af.py:224] 2022-01-20 18:33:55.131695: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.315746 140191300101952 af.py:224] 2022-01-20 18:33:55.131704: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.315787 140191300101952 af.py:224] 2022-01-20 18:33:55.131711: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315828 140191300101952 af.py:224] 2022-01-20 18:33:55.131718: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315869 140191300101952 af.py:224] 2022-01-20 18:33:55.131723: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.315909 140191300101952 af.py:224] 2022-01-20 18:33:55.131729: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.315950 140191300101952 af.py:224] 2022-01-20 18:33:55.131734: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.315990 140191300101952 af.py:224] 2022-01-20 18:33:55.131742: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.316031 140191300101952 af.py:224] 2022-01-20 18:33:55.131749: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316071 140191300101952 af.py:224] 2022-01-20 18:33:55.131755: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316111 140191300101952 af.py:224] 2022-01-20 18:33:55.131760: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.316157 140191300101952 af.py:224] 2022-01-20 18:33:55.131767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316198 140191300101952 af.py:224] 2022-01-20 18:33:55.131772: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.316238 140191300101952 af.py:224] 2022-01-20 18:33:55.131781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.316279 140191300101952 af.py:224] 2022-01-20 18:33:55.131788: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316320 140191300101952 af.py:224] 2022-01-20 18:33:55.131794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316361 140191300101952 af.py:224] 2022-01-20 18:33:55.131799: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.316405 140191300101952 af.py:224] 2022-01-20 18:33:55.131805: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316446 140191300101952 af.py:224] 2022-01-20 18:33:55.131810: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.316487 140191300101952 af.py:224] 2022-01-20 18:33:55.131819: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.316528 140191300101952 af.py:224] 2022-01-20 18:33:55.131826: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316569 140191300101952 af.py:224] 2022-01-20 18:33:55.131833: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316610 140191300101952 af.py:224] 2022-01-20 18:33:55.131838: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.316650 140191300101952 af.py:224] 2022-01-20 18:33:55.131844: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316690 140191300101952 af.py:224] 2022-01-20 18:33:55.131849: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.316731 140191300101952 af.py:224] 2022-01-20 18:33:55.131857: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.316773 140191300101952 af.py:224] 2022-01-20 18:33:55.131864: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316813 140191300101952 af.py:224] 2022-01-20 18:33:55.131870: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316853 140191300101952 af.py:224] 2022-01-20 18:33:55.131875: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.316894 140191300101952 af.py:224] 2022-01-20 18:33:55.131881: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.316935 140191300101952 af.py:224] 2022-01-20 18:33:55.131887: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.316976 140191300101952 af.py:224] 2022-01-20 18:33:55.131894: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.317016 140191300101952 af.py:224] 2022-01-20 18:33:55.131901: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317057 140191300101952 af.py:224] 2022-01-20 18:33:55.131908: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317100 140191300101952 af.py:224] 2022-01-20 18:33:55.131913: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.317142 140191300101952 af.py:224] 2022-01-20 18:33:55.131920: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317182 140191300101952 af.py:224] 2022-01-20 18:33:55.131925: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.317222 140191300101952 af.py:224] 2022-01-20 18:33:55.131933: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.317262 140191300101952 af.py:224] 2022-01-20 18:33:55.131940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317316 140191300101952 af.py:224] 2022-01-20 18:33:55.131947: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317358 140191300101952 af.py:224] 2022-01-20 18:33:55.131952: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.317398 140191300101952 af.py:224] 2022-01-20 18:33:55.131958: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317439 140191300101952 af.py:224] 2022-01-20 18:33:55.131964: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.317479 140191300101952 af.py:224] 2022-01-20 18:33:55.131972: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.317521 140191300101952 af.py:224] 2022-01-20 18:33:55.131980: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317561 140191300101952 af.py:224] 2022-01-20 18:33:55.131986: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317602 140191300101952 af.py:224] 2022-01-20 18:33:55.131991: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.317642 140191300101952 af.py:224] 2022-01-20 18:33:55.131998: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317683 140191300101952 af.py:224] 2022-01-20 18:33:55.132003: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.317724 140191300101952 af.py:224] 2022-01-20 18:33:55.132012: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.317768 140191300101952 af.py:224] 2022-01-20 18:33:55.132019: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317809 140191300101952 af.py:224] 2022-01-20 18:33:55.132025: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317850 140191300101952 af.py:224] 2022-01-20 18:33:55.132030: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.317891 140191300101952 af.py:224] 2022-01-20 18:33:55.132037: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.317931 140191300101952 af.py:224] 2022-01-20 18:33:55.132042: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.317972 140191300101952 af.py:224] 2022-01-20 18:33:55.132051: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.318013 140191300101952 af.py:224] 2022-01-20 18:33:55.132059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318053 140191300101952 af.py:224] 2022-01-20 18:33:55.132066: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318095 140191300101952 af.py:224] 2022-01-20 18:33:55.132071: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.318135 140191300101952 af.py:224] 2022-01-20 18:33:55.132077: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318176 140191300101952 af.py:224] 2022-01-20 18:33:55.132082: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.318217 140191300101952 af.py:224] 2022-01-20 18:33:55.132091: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.318258 140191300101952 af.py:224] 2022-01-20 18:33:55.132098: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318298 140191300101952 af.py:224] 2022-01-20 18:33:55.132104: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318339 140191300101952 af.py:224] 2022-01-20 18:33:55.132109: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.318379 140191300101952 af.py:224] 2022-01-20 18:33:55.132121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318419 140191300101952 af.py:224] 2022-01-20 18:33:55.132127: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.318463 140191300101952 af.py:224] 2022-01-20 18:33:55.132136: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.318504 140191300101952 af.py:224] 2022-01-20 18:33:55.132144: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318545 140191300101952 af.py:224] 2022-01-20 18:33:55.132150: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318585 140191300101952 af.py:224] 2022-01-20 18:33:55.132155: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.318626 140191300101952 af.py:224] 2022-01-20 18:33:55.132162: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318667 140191300101952 af.py:224] 2022-01-20 18:33:55.132167: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.318707 140191300101952 af.py:224] 2022-01-20 18:33:55.132176: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.318747 140191300101952 af.py:224] 2022-01-20 18:33:55.132183: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318788 140191300101952 af.py:224] 2022-01-20 18:33:55.132189: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318829 140191300101952 af.py:224] 2022-01-20 18:33:55.132194: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.318870 140191300101952 af.py:224] 2022-01-20 18:33:55.132200: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.318923 140191300101952 af.py:224] 2022-01-20 18:33:55.132205: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.318964 140191300101952 af.py:224] 2022-01-20 18:33:55.132213: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.319005 140191300101952 af.py:224] 2022-01-20 18:33:55.132219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319046 140191300101952 af.py:224] 2022-01-20 18:33:55.132226: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319087 140191300101952 af.py:224] 2022-01-20 18:33:55.132231: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.319128 140191300101952 af.py:224] 2022-01-20 18:33:55.132237: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319172 140191300101952 af.py:224] 2022-01-20 18:33:55.132243: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.319213 140191300101952 af.py:224] 2022-01-20 18:33:55.132251: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.319253 140191300101952 af.py:224] 2022-01-20 18:33:55.132258: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319294 140191300101952 af.py:224] 2022-01-20 18:33:55.132264: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319335 140191300101952 af.py:224] 2022-01-20 18:33:55.132270: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.319376 140191300101952 af.py:224] 2022-01-20 18:33:55.132275: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319416 140191300101952 af.py:224] 2022-01-20 18:33:55.132280: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.319456 140191300101952 af.py:224] 2022-01-20 18:33:55.132288: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.319496 140191300101952 af.py:224] 2022-01-20 18:33:55.132296: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319536 140191300101952 af.py:224] 2022-01-20 18:33:55.132302: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319577 140191300101952 af.py:224] 2022-01-20 18:33:55.132307: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.319617 140191300101952 af.py:224] 2022-01-20 18:33:55.132314: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319658 140191300101952 af.py:224] 2022-01-20 18:33:55.132319: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.319698 140191300101952 af.py:224] 2022-01-20 18:33:55.132327: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.319738 140191300101952 af.py:224] 2022-01-20 18:33:55.132335: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319778 140191300101952 af.py:224] 2022-01-20 18:33:55.132341: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319819 140191300101952 af.py:224] 2022-01-20 18:33:55.132346: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.319863 140191300101952 af.py:224] 2022-01-20 18:33:55.132352: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.319904 140191300101952 af.py:224] 2022-01-20 18:33:55.132357: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.319945 140191300101952 af.py:224] 2022-01-20 18:33:55.132366: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.319985 140191300101952 af.py:224] 2022-01-20 18:33:55.132372: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320026 140191300101952 af.py:224] 2022-01-20 18:33:55.132378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320067 140191300101952 af.py:224] 2022-01-20 18:33:55.132384: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.320107 140191300101952 af.py:224] 2022-01-20 18:33:55.132390: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320152 140191300101952 af.py:224] 2022-01-20 18:33:55.132395: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.320193 140191300101952 af.py:224] 2022-01-20 18:33:55.132405: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.320234 140191300101952 af.py:224] 2022-01-20 18:33:55.132412: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320274 140191300101952 af.py:224] 2022-01-20 18:33:55.132418: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320315 140191300101952 af.py:224] 2022-01-20 18:33:55.132423: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.320356 140191300101952 af.py:224] 2022-01-20 18:33:55.132429: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320397 140191300101952 af.py:224] 2022-01-20 18:33:55.132434: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.320437 140191300101952 af.py:224] 2022-01-20 18:33:55.132442: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.320478 140191300101952 af.py:224] 2022-01-20 18:33:55.132449: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320535 140191300101952 af.py:224] 2022-01-20 18:33:55.132455: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320606 140191300101952 af.py:224] 2022-01-20 18:33:55.132460: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.320648 140191300101952 af.py:224] 2022-01-20 18:33:55.132467: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320703 140191300101952 af.py:224] 2022-01-20 18:33:55.132473: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.320763 140191300101952 af.py:224] 2022-01-20 18:33:55.132481: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.320804 140191300101952 af.py:224] 2022-01-20 18:33:55.132487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320878 140191300101952 af.py:224] 2022-01-20 18:33:55.132493: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.320934 140191300101952 af.py:224] 2022-01-20 18:33:55.132499: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.320975 140191300101952 af.py:224] 2022-01-20 18:33:55.132505: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321015 140191300101952 af.py:224] 2022-01-20 18:33:55.132510: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.321056 140191300101952 af.py:224] 2022-01-20 18:33:55.132518: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.321097 140191300101952 af.py:224] 2022-01-20 18:33:55.132524: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321137 140191300101952 af.py:224] 2022-01-20 18:33:55.132531: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321178 140191300101952 af.py:224] 2022-01-20 18:33:55.132536: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.321219 140191300101952 af.py:224] 2022-01-20 18:33:55.132542: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321260 140191300101952 af.py:224] 2022-01-20 18:33:55.132547: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.321300 140191300101952 af.py:224] 2022-01-20 18:33:55.132556: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.321343 140191300101952 af.py:224] 2022-01-20 18:33:55.132562: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321385 140191300101952 af.py:224] 2022-01-20 18:33:55.132568: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321426 140191300101952 af.py:224] 2022-01-20 18:33:55.132573: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.321467 140191300101952 af.py:224] 2022-01-20 18:33:55.132579: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321508 140191300101952 af.py:224] 2022-01-20 18:33:55.132584: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.321548 140191300101952 af.py:224] 2022-01-20 18:33:55.132593: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.321589 140191300101952 af.py:224] 2022-01-20 18:33:55.132600: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321630 140191300101952 af.py:224] 2022-01-20 18:33:55.132607: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321671 140191300101952 af.py:224] 2022-01-20 18:33:55.132612: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.321712 140191300101952 af.py:224] 2022-01-20 18:33:55.132619: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321753 140191300101952 af.py:224] 2022-01-20 18:33:55.132624: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.321793 140191300101952 af.py:224] 2022-01-20 18:33:55.132632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.321834 140191300101952 af.py:224] 2022-01-20 18:33:55.132638: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321874 140191300101952 af.py:224] 2022-01-20 18:33:55.132644: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321915 140191300101952 af.py:224] 2022-01-20 18:33:55.132650: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.321955 140191300101952 af.py:224] 2022-01-20 18:33:55.132655: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.321995 140191300101952 af.py:224] 2022-01-20 18:33:55.132661: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.322039 140191300101952 af.py:224] 2022-01-20 18:33:55.132669: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.322080 140191300101952 af.py:224] 2022-01-20 18:33:55.132678: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322121 140191300101952 af.py:224] 2022-01-20 18:33:55.132684: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322162 140191300101952 af.py:224] 2022-01-20 18:33:55.132689: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.322203 140191300101952 af.py:224] 2022-01-20 18:33:55.132695: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322257 140191300101952 af.py:224] 2022-01-20 18:33:55.132701: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.322298 140191300101952 af.py:224] 2022-01-20 18:33:55.132709: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.322340 140191300101952 af.py:224] 2022-01-20 18:33:55.132716: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322381 140191300101952 af.py:224] 2022-01-20 18:33:55.132722: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322422 140191300101952 af.py:224] 2022-01-20 18:33:55.132727: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.322462 140191300101952 af.py:224] 2022-01-20 18:33:55.132734: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322503 140191300101952 af.py:224] 2022-01-20 18:33:55.132739: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.322544 140191300101952 af.py:224] 2022-01-20 18:33:55.132747: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.322585 140191300101952 af.py:224] 2022-01-20 18:33:55.132754: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322626 140191300101952 af.py:224] 2022-01-20 18:33:55.132760: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322666 140191300101952 af.py:224] 2022-01-20 18:33:55.132765: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.322707 140191300101952 af.py:224] 2022-01-20 18:33:55.132772: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322751 140191300101952 af.py:224] 2022-01-20 18:33:55.132777: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.322792 140191300101952 af.py:224] 2022-01-20 18:33:55.132785: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.322832 140191300101952 af.py:224] 2022-01-20 18:33:55.132792: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322873 140191300101952 af.py:224] 2022-01-20 18:33:55.132799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322914 140191300101952 af.py:224] 2022-01-20 18:33:55.132804: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.322954 140191300101952 af.py:224] 2022-01-20 18:33:55.132810: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.322995 140191300101952 af.py:224] 2022-01-20 18:33:55.132815: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.323035 140191300101952 af.py:224] 2022-01-20 18:33:55.132824: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.323076 140191300101952 af.py:224] 2022-01-20 18:33:55.132831: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323117 140191300101952 af.py:224] 2022-01-20 18:33:55.132837: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323157 140191300101952 af.py:224] 2022-01-20 18:33:55.132842: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.323198 140191300101952 af.py:224] 2022-01-20 18:33:55.132849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323239 140191300101952 af.py:224] 2022-01-20 18:33:55.132854: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.323279 140191300101952 af.py:224] 2022-01-20 18:33:55.132862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.323320 140191300101952 af.py:224] 2022-01-20 18:33:55.132869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323361 140191300101952 af.py:224] 2022-01-20 18:33:55.132875: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323406 140191300101952 af.py:224] 2022-01-20 18:33:55.132880: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.323447 140191300101952 af.py:224] 2022-01-20 18:33:55.132886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323488 140191300101952 af.py:224] 2022-01-20 18:33:55.132891: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.323529 140191300101952 af.py:224] 2022-01-20 18:33:55.132900: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.323569 140191300101952 af.py:224] 2022-01-20 18:33:55.132907: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323609 140191300101952 af.py:224] 2022-01-20 18:33:55.132913: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323650 140191300101952 af.py:224] 2022-01-20 18:33:55.132918: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.323692 140191300101952 af.py:224] 2022-01-20 18:33:55.132924: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323733 140191300101952 af.py:224] 2022-01-20 18:33:55.132930: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.323773 140191300101952 af.py:224] 2022-01-20 18:33:55.132937: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.323814 140191300101952 af.py:224] 2022-01-20 18:33:55.132944: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323868 140191300101952 af.py:224] 2022-01-20 18:33:55.132950: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323910 140191300101952 af.py:224] 2022-01-20 18:33:55.132956: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.323951 140191300101952 af.py:224] 2022-01-20 18:33:55.132962: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.323992 140191300101952 af.py:224] 2022-01-20 18:33:55.132967: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.324033 140191300101952 af.py:224] 2022-01-20 18:33:55.132976: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.324074 140191300101952 af.py:224] 2022-01-20 18:33:55.132983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324140 140191300101952 af.py:224] 2022-01-20 18:33:55.132989: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324200 140191300101952 af.py:224] 2022-01-20 18:33:55.132994: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.324240 140191300101952 af.py:224] 2022-01-20 18:33:55.133001: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324280 140191300101952 af.py:224] 2022-01-20 18:33:55.133006: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.324321 140191300101952 af.py:224] 2022-01-20 18:33:55.133014: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.324362 140191300101952 af.py:224] 2022-01-20 18:33:55.133021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324403 140191300101952 af.py:224] 2022-01-20 18:33:55.133028: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324443 140191300101952 af.py:224] 2022-01-20 18:33:55.133033: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.324484 140191300101952 af.py:224] 2022-01-20 18:33:55.133039: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324524 140191300101952 af.py:224] 2022-01-20 18:33:55.133044: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.324565 140191300101952 af.py:224] 2022-01-20 18:33:55.133053: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.324604 140191300101952 af.py:224] 2022-01-20 18:33:55.133060: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324644 140191300101952 af.py:224] 2022-01-20 18:33:55.133066: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324685 140191300101952 af.py:224] 2022-01-20 18:33:55.133071: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.324726 140191300101952 af.py:224] 2022-01-20 18:33:55.133078: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324766 140191300101952 af.py:224] 2022-01-20 18:33:55.133083: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.324806 140191300101952 af.py:224] 2022-01-20 18:33:55.133091: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.324850 140191300101952 af.py:224] 2022-01-20 18:33:55.133097: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324891 140191300101952 af.py:224] 2022-01-20 18:33:55.133104: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.324931 140191300101952 af.py:224] 2022-01-20 18:33:55.133109: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.324971 140191300101952 af.py:224] 2022-01-20 18:33:55.133116: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325012 140191300101952 af.py:224] 2022-01-20 18:33:55.133121: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.325053 140191300101952 af.py:224] 2022-01-20 18:33:55.133129: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.325093 140191300101952 af.py:224] 2022-01-20 18:33:55.133135: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325134 140191300101952 af.py:224] 2022-01-20 18:33:55.133142: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325174 140191300101952 af.py:224] 2022-01-20 18:33:55.133147: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.325215 140191300101952 af.py:224] 2022-01-20 18:33:55.133154: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325256 140191300101952 af.py:224] 2022-01-20 18:33:55.133159: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.325297 140191300101952 af.py:224] 2022-01-20 18:33:55.133167: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.325337 140191300101952 af.py:224] 2022-01-20 18:33:55.133175: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325378 140191300101952 af.py:224] 2022-01-20 18:33:55.133181: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325418 140191300101952 af.py:224] 2022-01-20 18:33:55.133186: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.325459 140191300101952 af.py:224] 2022-01-20 18:33:55.133193: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325499 140191300101952 af.py:224] 2022-01-20 18:33:55.133198: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.325556 140191300101952 af.py:224] 2022-01-20 18:33:55.133207: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.325598 140191300101952 af.py:224] 2022-01-20 18:33:55.133213: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325639 140191300101952 af.py:224] 2022-01-20 18:33:55.133219: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325680 140191300101952 af.py:224] 2022-01-20 18:33:55.133225: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.325720 140191300101952 af.py:224] 2022-01-20 18:33:55.133231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325761 140191300101952 af.py:224] 2022-01-20 18:33:55.133236: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.325801 140191300101952 af.py:224] 2022-01-20 18:33:55.133244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.325842 140191300101952 af.py:224] 2022-01-20 18:33:55.133251: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325883 140191300101952 af.py:224] 2022-01-20 18:33:55.133257: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.325923 140191300101952 af.py:224] 2022-01-20 18:33:55.133262: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.325963 140191300101952 af.py:224] 2022-01-20 18:33:55.133268: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326004 140191300101952 af.py:224] 2022-01-20 18:33:55.133274: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.326045 140191300101952 af.py:224] 2022-01-20 18:33:55.133282: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.326085 140191300101952 af.py:224] 2022-01-20 18:33:55.133289: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326126 140191300101952 af.py:224] 2022-01-20 18:33:55.133295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326167 140191300101952 af.py:224] 2022-01-20 18:33:55.133300: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.326207 140191300101952 af.py:224] 2022-01-20 18:33:55.133306: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326251 140191300101952 af.py:224] 2022-01-20 18:33:55.133311: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.326293 140191300101952 af.py:224] 2022-01-20 18:33:55.133319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.326333 140191300101952 af.py:224] 2022-01-20 18:33:55.133326: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326375 140191300101952 af.py:224] 2022-01-20 18:33:55.133332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326416 140191300101952 af.py:224] 2022-01-20 18:33:55.133336: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.326456 140191300101952 af.py:224] 2022-01-20 18:33:55.133343: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326497 140191300101952 af.py:224] 2022-01-20 18:33:55.133348: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.326537 140191300101952 af.py:224] 2022-01-20 18:33:55.133357: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.326577 140191300101952 af.py:224] 2022-01-20 18:33:55.133363: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326617 140191300101952 af.py:224] 2022-01-20 18:33:55.133369: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326658 140191300101952 af.py:224] 2022-01-20 18:33:55.133374: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.326699 140191300101952 af.py:224] 2022-01-20 18:33:55.133381: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326740 140191300101952 af.py:224] 2022-01-20 18:33:55.133386: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.326781 140191300101952 af.py:224] 2022-01-20 18:33:55.133394: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.326821 140191300101952 af.py:224] 2022-01-20 18:33:55.133401: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326862 140191300101952 af.py:224] 2022-01-20 18:33:55.133407: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326905 140191300101952 af.py:224] 2022-01-20 18:33:55.133411: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.326945 140191300101952 af.py:224] 2022-01-20 18:33:55.133417: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.326986 140191300101952 af.py:224] 2022-01-20 18:33:55.133423: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.327026 140191300101952 af.py:224] 2022-01-20 18:33:55.133431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.327067 140191300101952 af.py:224] 2022-01-20 18:33:55.133438: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327107 140191300101952 af.py:224] 2022-01-20 18:33:55.133444: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327163 140191300101952 af.py:224] 2022-01-20 18:33:55.133449: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.327224 140191300101952 af.py:224] 2022-01-20 18:33:55.133456: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327286 140191300101952 af.py:224] 2022-01-20 18:33:55.133461: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.327326 140191300101952 af.py:224] 2022-01-20 18:33:55.133470: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.327383 140191300101952 af.py:224] 2022-01-20 18:33:55.133476: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327424 140191300101952 af.py:224] 2022-01-20 18:33:55.133482: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327465 140191300101952 af.py:224] 2022-01-20 18:33:55.133487: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.327521 140191300101952 af.py:224] 2022-01-20 18:33:55.133493: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327560 140191300101952 af.py:224] 2022-01-20 18:33:55.133498: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.327601 140191300101952 af.py:224] 2022-01-20 18:33:55.133506: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.327642 140191300101952 af.py:224] 2022-01-20 18:33:55.133513: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327686 140191300101952 af.py:224] 2022-01-20 18:33:55.133519: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327726 140191300101952 af.py:224] 2022-01-20 18:33:55.133524: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.327766 140191300101952 af.py:224] 2022-01-20 18:33:55.133531: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327806 140191300101952 af.py:224] 2022-01-20 18:33:55.133536: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.327847 140191300101952 af.py:224] 2022-01-20 18:33:55.133544: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.327887 140191300101952 af.py:224] 2022-01-20 18:33:55.133551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327927 140191300101952 af.py:224] 2022-01-20 18:33:55.133558: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.327968 140191300101952 af.py:224] 2022-01-20 18:33:55.133563: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.328008 140191300101952 af.py:224] 2022-01-20 18:33:55.133569: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328049 140191300101952 af.py:224] 2022-01-20 18:33:55.133575: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.328089 140191300101952 af.py:224] 2022-01-20 18:33:55.133583: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.328167 140191300101952 af.py:224] 2022-01-20 18:33:55.133590: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328209 140191300101952 af.py:224] 2022-01-20 18:33:55.133597: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328250 140191300101952 af.py:224] 2022-01-20 18:33:55.133602: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.328290 140191300101952 af.py:224] 2022-01-20 18:33:55.133609: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328330 140191300101952 af.py:224] 2022-01-20 18:33:55.133614: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.328371 140191300101952 af.py:224] 2022-01-20 18:33:55.133622: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.328415 140191300101952 af.py:224] 2022-01-20 18:33:55.133629: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328456 140191300101952 af.py:224] 2022-01-20 18:33:55.133636: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328497 140191300101952 af.py:224] 2022-01-20 18:33:55.133640: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.328538 140191300101952 af.py:224] 2022-01-20 18:33:55.133647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328578 140191300101952 af.py:224] 2022-01-20 18:33:55.133652: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.328619 140191300101952 af.py:224] 2022-01-20 18:33:55.133660: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.328659 140191300101952 af.py:224] 2022-01-20 18:33:55.133666: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328700 140191300101952 af.py:224] 2022-01-20 18:33:55.133673: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328740 140191300101952 af.py:224] 2022-01-20 18:33:55.133678: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.328781 140191300101952 af.py:224] 2022-01-20 18:33:55.133685: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328821 140191300101952 af.py:224] 2022-01-20 18:33:55.133689: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.328876 140191300101952 af.py:224] 2022-01-20 18:33:55.133698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.328918 140191300101952 af.py:224] 2022-01-20 18:33:55.133705: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.328959 140191300101952 af.py:224] 2022-01-20 18:33:55.133711: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329000 140191300101952 af.py:224] 2022-01-20 18:33:55.133716: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.329041 140191300101952 af.py:224] 2022-01-20 18:33:55.133722: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329082 140191300101952 af.py:224] 2022-01-20 18:33:55.133727: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.329127 140191300101952 af.py:224] 2022-01-20 18:33:55.133736: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.329168 140191300101952 af.py:224] 2022-01-20 18:33:55.133743: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329209 140191300101952 af.py:224] 2022-01-20 18:33:55.133750: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329249 140191300101952 af.py:224] 2022-01-20 18:33:55.133755: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.329289 140191300101952 af.py:224] 2022-01-20 18:33:55.133762: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329329 140191300101952 af.py:224] 2022-01-20 18:33:55.133767: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.329370 140191300101952 af.py:224] 2022-01-20 18:33:55.133775: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.329411 140191300101952 af.py:224] 2022-01-20 18:33:55.133782: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329451 140191300101952 af.py:224] 2022-01-20 18:33:55.133788: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329492 140191300101952 af.py:224] 2022-01-20 18:33:55.133792: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.329532 140191300101952 af.py:224] 2022-01-20 18:33:55.133799: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329573 140191300101952 af.py:224] 2022-01-20 18:33:55.133804: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.329613 140191300101952 af.py:224] 2022-01-20 18:33:55.133812: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.329653 140191300101952 af.py:224] 2022-01-20 18:33:55.133818: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329694 140191300101952 af.py:224] 2022-01-20 18:33:55.133824: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329734 140191300101952 af.py:224] 2022-01-20 18:33:55.133830: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.329775 140191300101952 af.py:224] 2022-01-20 18:33:55.133836: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329818 140191300101952 af.py:224] 2022-01-20 18:33:55.133841: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.329859 140191300101952 af.py:224] 2022-01-20 18:33:55.133849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.329899 140191300101952 af.py:224] 2022-01-20 18:33:55.133856: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329940 140191300101952 af.py:224] 2022-01-20 18:33:55.133862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.329981 140191300101952 af.py:224] 2022-01-20 18:33:55.133867: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.330021 140191300101952 af.py:224] 2022-01-20 18:33:55.133873: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330062 140191300101952 af.py:224] 2022-01-20 18:33:55.133878: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.330102 140191300101952 af.py:224] 2022-01-20 18:33:55.133886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.330142 140191300101952 af.py:224] 2022-01-20 18:33:55.133893: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330183 140191300101952 af.py:224] 2022-01-20 18:33:55.133900: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330223 140191300101952 af.py:224] 2022-01-20 18:33:55.133905: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.330264 140191300101952 af.py:224] 2022-01-20 18:33:55.133911: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330305 140191300101952 af.py:224] 2022-01-20 18:33:55.133916: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.330346 140191300101952 af.py:224] 2022-01-20 18:33:55.133925: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.330386 140191300101952 af.py:224] 2022-01-20 18:33:55.133933: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330427 140191300101952 af.py:224] 2022-01-20 18:33:55.133939: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330483 140191300101952 af.py:224] 2022-01-20 18:33:55.133943: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.330526 140191300101952 af.py:224] 2022-01-20 18:33:55.133950: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330567 140191300101952 af.py:224] 2022-01-20 18:33:55.133955: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.330607 140191300101952 af.py:224] 2022-01-20 18:33:55.133963: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.330648 140191300101952 af.py:224] 2022-01-20 18:33:55.133970: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330690 140191300101952 af.py:224] 2022-01-20 18:33:55.133977: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330731 140191300101952 af.py:224] 2022-01-20 18:33:55.133982: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.330772 140191300101952 af.py:224] 2022-01-20 18:33:55.133988: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330813 140191300101952 af.py:224] 2022-01-20 18:33:55.133994: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.330853 140191300101952 af.py:224] 2022-01-20 18:33:55.134002: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.330894 140191300101952 af.py:224] 2022-01-20 18:33:55.134009: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330934 140191300101952 af.py:224] 2022-01-20 18:33:55.134015: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.330975 140191300101952 af.py:224] 2022-01-20 18:33:55.134019: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.331015 140191300101952 af.py:224] 2022-01-20 18:33:55.134026: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331055 140191300101952 af.py:224] 2022-01-20 18:33:55.134031: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.331096 140191300101952 af.py:224] 2022-01-20 18:33:55.134038: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.331136 140191300101952 af.py:224] 2022-01-20 18:33:55.134045: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331179 140191300101952 af.py:224] 2022-01-20 18:33:55.134052: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331221 140191300101952 af.py:224] 2022-01-20 18:33:55.134057: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.331261 140191300101952 af.py:224] 2022-01-20 18:33:55.134063: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331301 140191300101952 af.py:224] 2022-01-20 18:33:55.134068: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.331341 140191300101952 af.py:224] 2022-01-20 18:33:55.134076: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.331382 140191300101952 af.py:224] 2022-01-20 18:33:55.134083: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331422 140191300101952 af.py:224] 2022-01-20 18:33:55.134088: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331462 140191300101952 af.py:224] 2022-01-20 18:33:55.134094: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.331502 140191300101952 af.py:224] 2022-01-20 18:33:55.134100: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331542 140191300101952 af.py:224] 2022-01-20 18:33:55.134105: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.331583 140191300101952 af.py:224] 2022-01-20 18:33:55.134113: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.331623 140191300101952 af.py:224] 2022-01-20 18:33:55.134120: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331664 140191300101952 af.py:224] 2022-01-20 18:33:55.134126: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331705 140191300101952 af.py:224] 2022-01-20 18:33:55.134131: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.331746 140191300101952 af.py:224] 2022-01-20 18:33:55.134138: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331787 140191300101952 af.py:224] 2022-01-20 18:33:55.134143: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.331828 140191300101952 af.py:224] 2022-01-20 18:33:55.134151: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.331871 140191300101952 af.py:224] 2022-01-20 18:33:55.134158: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331912 140191300101952 af.py:224] 2022-01-20 18:33:55.134164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.331953 140191300101952 af.py:224] 2022-01-20 18:33:55.134169: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.331994 140191300101952 af.py:224] 2022-01-20 18:33:55.134175: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332035 140191300101952 af.py:224] 2022-01-20 18:33:55.134180: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.332076 140191300101952 af.py:224] 2022-01-20 18:33:55.134188: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.332134 140191300101952 af.py:224] 2022-01-20 18:33:55.134194: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332177 140191300101952 af.py:224] 2022-01-20 18:33:55.134200: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332219 140191300101952 af.py:224] 2022-01-20 18:33:55.134205: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.332260 140191300101952 af.py:224] 2022-01-20 18:33:55.134211: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332301 140191300101952 af.py:224] 2022-01-20 18:33:55.134217: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.332341 140191300101952 af.py:224] 2022-01-20 18:33:55.134225: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.332382 140191300101952 af.py:224] 2022-01-20 18:33:55.134232: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332423 140191300101952 af.py:224] 2022-01-20 18:33:55.134238: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332465 140191300101952 af.py:224] 2022-01-20 18:33:55.134243: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.332504 140191300101952 af.py:224] 2022-01-20 18:33:55.134250: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332545 140191300101952 af.py:224] 2022-01-20 18:33:55.134255: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.332590 140191300101952 af.py:224] 2022-01-20 18:33:55.134263: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.332631 140191300101952 af.py:224] 2022-01-20 18:33:55.134270: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332672 140191300101952 af.py:224] 2022-01-20 18:33:55.134276: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332713 140191300101952 af.py:224] 2022-01-20 18:33:55.134281: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.332754 140191300101952 af.py:224] 2022-01-20 18:33:55.134287: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332794 140191300101952 af.py:224] 2022-01-20 18:33:55.134292: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.332834 140191300101952 af.py:224] 2022-01-20 18:33:55.134301: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.332875 140191300101952 af.py:224] 2022-01-20 18:33:55.134309: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332916 140191300101952 af.py:224] 2022-01-20 18:33:55.134315: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.332956 140191300101952 af.py:224] 2022-01-20 18:33:55.134320: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.332997 140191300101952 af.py:224] 2022-01-20 18:33:55.134327: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333038 140191300101952 af.py:224] 2022-01-20 18:33:55.134332: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.333078 140191300101952 af.py:224] 2022-01-20 18:33:55.134340: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.333119 140191300101952 af.py:224] 2022-01-20 18:33:55.134347: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333160 140191300101952 af.py:224] 2022-01-20 18:33:55.134353: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333201 140191300101952 af.py:224] 2022-01-20 18:33:55.134358: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.333241 140191300101952 af.py:224] 2022-01-20 18:33:55.134364: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333285 140191300101952 af.py:224] 2022-01-20 18:33:55.134369: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.333326 140191300101952 af.py:224] 2022-01-20 18:33:55.134378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.333367 140191300101952 af.py:224] 2022-01-20 18:33:55.134386: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333408 140191300101952 af.py:224] 2022-01-20 18:33:55.134392: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333448 140191300101952 af.py:224] 2022-01-20 18:33:55.134397: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.333488 140191300101952 af.py:224] 2022-01-20 18:33:55.134402: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333529 140191300101952 af.py:224] 2022-01-20 18:33:55.134407: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.333570 140191300101952 af.py:224] 2022-01-20 18:33:55.134416: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.333611 140191300101952 af.py:224] 2022-01-20 18:33:55.134423: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333651 140191300101952 af.py:224] 2022-01-20 18:33:55.134429: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333692 140191300101952 af.py:224] 2022-01-20 18:33:55.134434: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.333750 140191300101952 af.py:224] 2022-01-20 18:33:55.134441: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333792 140191300101952 af.py:224] 2022-01-20 18:33:55.134446: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.333858 140191300101952 af.py:224] 2022-01-20 18:33:55.134454: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.333900 140191300101952 af.py:224] 2022-01-20 18:33:55.134461: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.333941 140191300101952 af.py:224] 2022-01-20 18:33:55.134467: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334021 140191300101952 af.py:224] 2022-01-20 18:33:55.134472: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.334097 140191300101952 af.py:224] 2022-01-20 18:33:55.134494: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334154 140191300101952 af.py:224] 2022-01-20 18:33:55.134499: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.334195 140191300101952 af.py:224] 2022-01-20 18:33:55.134508: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.334235 140191300101952 af.py:224] 2022-01-20 18:33:55.134516: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334276 140191300101952 af.py:224] 2022-01-20 18:33:55.134522: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334317 140191300101952 af.py:224] 2022-01-20 18:33:55.134527: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.334357 140191300101952 af.py:224] 2022-01-20 18:33:55.134534: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334398 140191300101952 af.py:224] 2022-01-20 18:33:55.134539: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.334439 140191300101952 af.py:224] 2022-01-20 18:33:55.134547: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.334480 140191300101952 af.py:224] 2022-01-20 18:33:55.134554: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334520 140191300101952 af.py:224] 2022-01-20 18:33:55.134560: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334561 140191300101952 af.py:224] 2022-01-20 18:33:55.134565: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.334602 140191300101952 af.py:224] 2022-01-20 18:33:55.134571: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334644 140191300101952 af.py:224] 2022-01-20 18:33:55.134576: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.334684 140191300101952 af.py:224] 2022-01-20 18:33:55.134584: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.334725 140191300101952 af.py:224] 2022-01-20 18:33:55.134592: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334769 140191300101952 af.py:224] 2022-01-20 18:33:55.134598: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334810 140191300101952 af.py:224] 2022-01-20 18:33:55.134603: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.334850 140191300101952 af.py:224] 2022-01-20 18:33:55.134610: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.334891 140191300101952 af.py:224] 2022-01-20 18:33:55.134615: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.334932 140191300101952 af.py:224] 2022-01-20 18:33:55.134622: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.334973 140191300101952 af.py:224] 2022-01-20 18:33:55.134629: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335015 140191300101952 af.py:224] 2022-01-20 18:33:55.134635: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335055 140191300101952 af.py:224] 2022-01-20 18:33:55.134640: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.335096 140191300101952 af.py:224] 2022-01-20 18:33:55.134647: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335136 140191300101952 af.py:224] 2022-01-20 18:33:55.134652: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.335177 140191300101952 af.py:224] 2022-01-20 18:33:55.134660: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.335217 140191300101952 af.py:224] 2022-01-20 18:33:55.134667: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335258 140191300101952 af.py:224] 2022-01-20 18:33:55.134673: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335298 140191300101952 af.py:224] 2022-01-20 18:33:55.134678: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.335339 140191300101952 af.py:224] 2022-01-20 18:33:55.134685: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335380 140191300101952 af.py:224] 2022-01-20 18:33:55.134690: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.335421 140191300101952 af.py:224] 2022-01-20 18:33:55.134698: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.335480 140191300101952 af.py:224] 2022-01-20 18:33:55.134705: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335522 140191300101952 af.py:224] 2022-01-20 18:33:55.134712: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335564 140191300101952 af.py:224] 2022-01-20 18:33:55.134717: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.335604 140191300101952 af.py:224] 2022-01-20 18:33:55.134723: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335645 140191300101952 af.py:224] 2022-01-20 18:33:55.134728: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.335686 140191300101952 af.py:224] 2022-01-20 18:33:55.134737: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.335726 140191300101952 af.py:224] 2022-01-20 18:33:55.134744: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335767 140191300101952 af.py:224] 2022-01-20 18:33:55.134750: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335808 140191300101952 af.py:224] 2022-01-20 18:33:55.134755: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.335848 140191300101952 af.py:224] 2022-01-20 18:33:55.134761: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.335889 140191300101952 af.py:224] 2022-01-20 18:33:55.134766: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.335930 140191300101952 af.py:224] 2022-01-20 18:33:55.134774: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.335970 140191300101952 af.py:224] 2022-01-20 18:33:55.134781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336011 140191300101952 af.py:224] 2022-01-20 18:33:55.134787: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336051 140191300101952 af.py:224] 2022-01-20 18:33:55.134792: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.336092 140191300101952 af.py:224] 2022-01-20 18:33:55.134798: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336170 140191300101952 af.py:224] 2022-01-20 18:33:55.134803: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.336215 140191300101952 af.py:224] 2022-01-20 18:33:55.134811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.336256 140191300101952 af.py:224] 2022-01-20 18:33:55.134818: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336297 140191300101952 af.py:224] 2022-01-20 18:33:55.134824: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336337 140191300101952 af.py:224] 2022-01-20 18:33:55.134829: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.336378 140191300101952 af.py:224] 2022-01-20 18:33:55.134836: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336419 140191300101952 af.py:224] 2022-01-20 18:33:55.134841: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.336459 140191300101952 af.py:224] 2022-01-20 18:33:55.134848: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.336500 140191300101952 af.py:224] 2022-01-20 18:33:55.134855: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336540 140191300101952 af.py:224] 2022-01-20 18:33:55.134861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336581 140191300101952 af.py:224] 2022-01-20 18:33:55.134866: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.336621 140191300101952 af.py:224] 2022-01-20 18:33:55.134872: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336662 140191300101952 af.py:224] 2022-01-20 18:33:55.134878: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.336702 140191300101952 af.py:224] 2022-01-20 18:33:55.134886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.336743 140191300101952 af.py:224] 2022-01-20 18:33:55.134893: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336783 140191300101952 af.py:224] 2022-01-20 18:33:55.134899: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336825 140191300101952 af.py:224] 2022-01-20 18:33:55.134904: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.336865 140191300101952 af.py:224] 2022-01-20 18:33:55.134911: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.336909 140191300101952 af.py:224] 2022-01-20 18:33:55.134916: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.336950 140191300101952 af.py:224] 2022-01-20 18:33:55.134923: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.336991 140191300101952 af.py:224] 2022-01-20 18:33:55.134930: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337032 140191300101952 af.py:224] 2022-01-20 18:33:55.134936: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337073 140191300101952 af.py:224] 2022-01-20 18:33:55.134941: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.337127 140191300101952 af.py:224] 2022-01-20 18:33:55.134947: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337169 140191300101952 af.py:224] 2022-01-20 18:33:55.134952: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.337209 140191300101952 af.py:224] 2022-01-20 18:33:55.134959: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.337250 140191300101952 af.py:224] 2022-01-20 18:33:55.134966: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337291 140191300101952 af.py:224] 2022-01-20 18:33:55.134972: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337332 140191300101952 af.py:224] 2022-01-20 18:33:55.134977: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.337373 140191300101952 af.py:224] 2022-01-20 18:33:55.134983: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337413 140191300101952 af.py:224] 2022-01-20 18:33:55.134988: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.337454 140191300101952 af.py:224] 2022-01-20 18:33:55.134997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.337494 140191300101952 af.py:224] 2022-01-20 18:33:55.135004: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337534 140191300101952 af.py:224] 2022-01-20 18:33:55.135010: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337579 140191300101952 af.py:224] 2022-01-20 18:33:55.135015: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.337620 140191300101952 af.py:224] 2022-01-20 18:33:55.135022: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337660 140191300101952 af.py:224] 2022-01-20 18:33:55.135027: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.337701 140191300101952 af.py:224] 2022-01-20 18:33:55.135035: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.337741 140191300101952 af.py:224] 2022-01-20 18:33:55.135042: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337783 140191300101952 af.py:224] 2022-01-20 18:33:55.135048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337823 140191300101952 af.py:224] 2022-01-20 18:33:55.135053: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.337864 140191300101952 af.py:224] 2022-01-20 18:33:55.135059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.337904 140191300101952 af.py:224] 2022-01-20 18:33:55.135064: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.337944 140191300101952 af.py:224] 2022-01-20 18:33:55.135072: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.337985 140191300101952 af.py:224] 2022-01-20 18:33:55.135079: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338026 140191300101952 af.py:224] 2022-01-20 18:33:55.135085: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338066 140191300101952 af.py:224] 2022-01-20 18:33:55.135090: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.338106 140191300101952 af.py:224] 2022-01-20 18:33:55.135097: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338147 140191300101952 af.py:224] 2022-01-20 18:33:55.135102: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.338187 140191300101952 af.py:224] 2022-01-20 18:33:55.135110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.338228 140191300101952 af.py:224] 2022-01-20 18:33:55.135117: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338272 140191300101952 af.py:224] 2022-01-20 18:33:55.135124: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338312 140191300101952 af.py:224] 2022-01-20 18:33:55.135129: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.338352 140191300101952 af.py:224] 2022-01-20 18:33:55.135136: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338392 140191300101952 af.py:224] 2022-01-20 18:33:55.135141: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.338433 140191300101952 af.py:224] 2022-01-20 18:33:55.135148: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.338474 140191300101952 af.py:224] 2022-01-20 18:33:55.135155: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338514 140191300101952 af.py:224] 2022-01-20 18:33:55.135161: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338555 140191300101952 af.py:224] 2022-01-20 18:33:55.135166: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.338595 140191300101952 af.py:224] 2022-01-20 18:33:55.135173: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338635 140191300101952 af.py:224] 2022-01-20 18:33:55.135178: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.338676 140191300101952 af.py:224] 2022-01-20 18:33:55.135186: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.338730 140191300101952 af.py:224] 2022-01-20 18:33:55.135193: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338772 140191300101952 af.py:224] 2022-01-20 18:33:55.135199: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338812 140191300101952 af.py:224] 2022-01-20 18:33:55.135204: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.338852 140191300101952 af.py:224] 2022-01-20 18:33:55.135211: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.338893 140191300101952 af.py:224] 2022-01-20 18:33:55.135216: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.338933 140191300101952 af.py:224] 2022-01-20 18:33:55.135223: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.338977 140191300101952 af.py:224] 2022-01-20 18:33:55.135229: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339019 140191300101952 af.py:224] 2022-01-20 18:33:55.135236: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339060 140191300101952 af.py:224] 2022-01-20 18:33:55.135241: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.339101 140191300101952 af.py:224] 2022-01-20 18:33:55.135248: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339141 140191300101952 af.py:224] 2022-01-20 18:33:55.135253: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.339182 140191300101952 af.py:224] 2022-01-20 18:33:55.135261: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.339223 140191300101952 af.py:224] 2022-01-20 18:33:55.135268: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339264 140191300101952 af.py:224] 2022-01-20 18:33:55.135274: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339305 140191300101952 af.py:224] 2022-01-20 18:33:55.135279: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.339346 140191300101952 af.py:224] 2022-01-20 18:33:55.135285: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339387 140191300101952 af.py:224] 2022-01-20 18:33:55.135290: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.339428 140191300101952 af.py:224] 2022-01-20 18:33:55.135298: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.339469 140191300101952 af.py:224] 2022-01-20 18:33:55.135304: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339510 140191300101952 af.py:224] 2022-01-20 18:33:55.135311: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339551 140191300101952 af.py:224] 2022-01-20 18:33:55.135316: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.339592 140191300101952 af.py:224] 2022-01-20 18:33:55.135322: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339633 140191300101952 af.py:224] 2022-01-20 18:33:55.135327: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.339677 140191300101952 af.py:224] 2022-01-20 18:33:55.135336: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.339718 140191300101952 af.py:224] 2022-01-20 18:33:55.135342: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339759 140191300101952 af.py:224] 2022-01-20 18:33:55.135349: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339800 140191300101952 af.py:224] 2022-01-20 18:33:55.135354: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.339841 140191300101952 af.py:224] 2022-01-20 18:33:55.135361: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.339882 140191300101952 af.py:224] 2022-01-20 18:33:55.135366: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.339922 140191300101952 af.py:224] 2022-01-20 18:33:55.135374: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.339962 140191300101952 af.py:224] 2022-01-20 18:33:55.135382: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340003 140191300101952 af.py:224] 2022-01-20 18:33:55.135388: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340044 140191300101952 af.py:224] 2022-01-20 18:33:55.135393: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.340084 140191300101952 af.py:224] 2022-01-20 18:33:55.135400: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340128 140191300101952 af.py:224] 2022-01-20 18:33:55.135405: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.340170 140191300101952 af.py:224] 2022-01-20 18:33:55.135413: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.340211 140191300101952 af.py:224] 2022-01-20 18:33:55.135420: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340251 140191300101952 af.py:224] 2022-01-20 18:33:55.135426: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340292 140191300101952 af.py:224] 2022-01-20 18:33:55.135431: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.340348 140191300101952 af.py:224] 2022-01-20 18:33:55.135438: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340395 140191300101952 af.py:224] 2022-01-20 18:33:55.135443: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.340436 140191300101952 af.py:224] 2022-01-20 18:33:55.135450: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.340477 140191300101952 af.py:224] 2022-01-20 18:33:55.135458: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340518 140191300101952 af.py:224] 2022-01-20 18:33:55.135464: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340577 140191300101952 af.py:224] 2022-01-20 18:33:55.135470: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.340634 140191300101952 af.py:224] 2022-01-20 18:33:55.135476: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340675 140191300101952 af.py:224] 2022-01-20 18:33:55.135481: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.340716 140191300101952 af.py:224] 2022-01-20 18:33:55.135489: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.340756 140191300101952 af.py:224] 2022-01-20 18:33:55.135496: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340797 140191300101952 af.py:224] 2022-01-20 18:33:55.135502: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340838 140191300101952 af.py:224] 2022-01-20 18:33:55.135507: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.340878 140191300101952 af.py:224] 2022-01-20 18:33:55.135513: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.340919 140191300101952 af.py:224] 2022-01-20 18:33:55.135518: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.340959 140191300101952 af.py:224] 2022-01-20 18:33:55.135526: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.340999 140191300101952 af.py:224] 2022-01-20 18:33:55.135532: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341040 140191300101952 af.py:224] 2022-01-20 18:33:55.135538: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341084 140191300101952 af.py:224] 2022-01-20 18:33:55.135544: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.341125 140191300101952 af.py:224] 2022-01-20 18:33:55.135550: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341165 140191300101952 af.py:224] 2022-01-20 18:33:55.135555: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.341206 140191300101952 af.py:224] 2022-01-20 18:33:55.135563: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.341247 140191300101952 af.py:224] 2022-01-20 18:33:55.135570: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341287 140191300101952 af.py:224] 2022-01-20 18:33:55.135577: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341327 140191300101952 af.py:224] 2022-01-20 18:33:55.135581: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.341367 140191300101952 af.py:224] 2022-01-20 18:33:55.135588: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341408 140191300101952 af.py:224] 2022-01-20 18:33:55.135593: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.341448 140191300101952 af.py:224] 2022-01-20 18:33:55.135601: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.341488 140191300101952 af.py:224] 2022-01-20 18:33:55.135608: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341529 140191300101952 af.py:224] 2022-01-20 18:33:55.135614: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341570 140191300101952 af.py:224] 2022-01-20 18:33:55.135619: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.341610 140191300101952 af.py:224] 2022-01-20 18:33:55.135626: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341651 140191300101952 af.py:224] 2022-01-20 18:33:55.135631: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.341691 140191300101952 af.py:224] 2022-01-20 18:33:55.135638: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.341732 140191300101952 af.py:224] 2022-01-20 18:33:55.135646: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341775 140191300101952 af.py:224] 2022-01-20 18:33:55.135653: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341816 140191300101952 af.py:224] 2022-01-20 18:33:55.135658: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.341856 140191300101952 af.py:224] 2022-01-20 18:33:55.135664: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.341897 140191300101952 af.py:224] 2022-01-20 18:33:55.135669: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.341938 140191300101952 af.py:224] 2022-01-20 18:33:55.135677: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.341978 140191300101952 af.py:224] 2022-01-20 18:33:55.135683: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342033 140191300101952 af.py:224] 2022-01-20 18:33:55.135689: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342075 140191300101952 af.py:224] 2022-01-20 18:33:55.135694: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.342117 140191300101952 af.py:224] 2022-01-20 18:33:55.135701: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342158 140191300101952 af.py:224] 2022-01-20 18:33:55.135706: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.342198 140191300101952 af.py:224] 2022-01-20 18:33:55.135714: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.342239 140191300101952 af.py:224] 2022-01-20 18:33:55.135721: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342280 140191300101952 af.py:224] 2022-01-20 18:33:55.135727: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342321 140191300101952 af.py:224] 2022-01-20 18:33:55.135732: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.342362 140191300101952 af.py:224] 2022-01-20 18:33:55.135738: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342403 140191300101952 af.py:224] 2022-01-20 18:33:55.135743: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.342443 140191300101952 af.py:224] 2022-01-20 18:33:55.135750: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.342487 140191300101952 af.py:224] 2022-01-20 18:33:55.135757: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342528 140191300101952 af.py:224] 2022-01-20 18:33:55.135763: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342569 140191300101952 af.py:224] 2022-01-20 18:33:55.135768: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.342610 140191300101952 af.py:224] 2022-01-20 18:33:55.135775: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342650 140191300101952 af.py:224] 2022-01-20 18:33:55.135780: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.342691 140191300101952 af.py:224] 2022-01-20 18:33:55.135787: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.342732 140191300101952 af.py:224] 2022-01-20 18:33:55.135794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342772 140191300101952 af.py:224] 2022-01-20 18:33:55.135800: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342813 140191300101952 af.py:224] 2022-01-20 18:33:55.135805: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.342854 140191300101952 af.py:224] 2022-01-20 18:33:55.135811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.342894 140191300101952 af.py:224] 2022-01-20 18:33:55.135817: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.342935 140191300101952 af.py:224] 2022-01-20 18:33:55.135824: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.342976 140191300101952 af.py:224] 2022-01-20 18:33:55.135832: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343016 140191300101952 af.py:224] 2022-01-20 18:33:55.135838: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343057 140191300101952 af.py:224] 2022-01-20 18:33:55.135843: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.343098 140191300101952 af.py:224] 2022-01-20 18:33:55.135849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343138 140191300101952 af.py:224] 2022-01-20 18:33:55.135854: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.343182 140191300101952 af.py:224] 2022-01-20 18:33:55.135862: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.343223 140191300101952 af.py:224] 2022-01-20 18:33:55.135869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343263 140191300101952 af.py:224] 2022-01-20 18:33:55.135875: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343304 140191300101952 af.py:224] 2022-01-20 18:33:55.135879: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.343345 140191300101952 af.py:224] 2022-01-20 18:33:55.135886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343386 140191300101952 af.py:224] 2022-01-20 18:33:55.135891: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.343426 140191300101952 af.py:224] 2022-01-20 18:33:55.135898: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.343467 140191300101952 af.py:224] 2022-01-20 18:33:55.135905: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343507 140191300101952 af.py:224] 2022-01-20 18:33:55.135912: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343548 140191300101952 af.py:224] 2022-01-20 18:33:55.135917: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.343588 140191300101952 af.py:224] 2022-01-20 18:33:55.135923: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343641 140191300101952 af.py:224] 2022-01-20 18:33:55.135928: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.343683 140191300101952 af.py:224] 2022-01-20 18:33:55.135936: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.343724 140191300101952 af.py:224] 2022-01-20 18:33:55.135943: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343764 140191300101952 af.py:224] 2022-01-20 18:33:55.135949: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343805 140191300101952 af.py:224] 2022-01-20 18:33:55.135953: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.343846 140191300101952 af.py:224] 2022-01-20 18:33:55.135959: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.343890 140191300101952 af.py:224] 2022-01-20 18:33:55.135964: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.343930 140191300101952 af.py:224] 2022-01-20 18:33:55.135972: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.343970 140191300101952 af.py:224] 2022-01-20 18:33:55.135979: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344011 140191300101952 af.py:224] 2022-01-20 18:33:55.135985: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344052 140191300101952 af.py:224] 2022-01-20 18:33:55.135990: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.344092 140191300101952 af.py:224] 2022-01-20 18:33:55.135997: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344137 140191300101952 af.py:224] 2022-01-20 18:33:55.136002: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.344178 140191300101952 af.py:224] 2022-01-20 18:33:55.136010: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.344219 140191300101952 af.py:224] 2022-01-20 18:33:55.136017: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344260 140191300101952 af.py:224] 2022-01-20 18:33:55.136023: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344300 140191300101952 af.py:224] 2022-01-20 18:33:55.136028: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.344341 140191300101952 af.py:224] 2022-01-20 18:33:55.136035: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344382 140191300101952 af.py:224] 2022-01-20 18:33:55.136040: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.344422 140191300101952 af.py:224] 2022-01-20 18:33:55.136048: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.344463 140191300101952 af.py:224] 2022-01-20 18:33:55.136054: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344503 140191300101952 af.py:224] 2022-01-20 18:33:55.136060: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344547 140191300101952 af.py:224] 2022-01-20 18:33:55.136066: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.344589 140191300101952 af.py:224] 2022-01-20 18:33:55.136072: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344630 140191300101952 af.py:224] 2022-01-20 18:33:55.136077: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.344671 140191300101952 af.py:224] 2022-01-20 18:33:55.136085: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.344712 140191300101952 af.py:224] 2022-01-20 18:33:55.136091: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344752 140191300101952 af.py:224] 2022-01-20 18:33:55.136097: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344792 140191300101952 af.py:224] 2022-01-20 18:33:55.136101: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.344833 140191300101952 af.py:224] 2022-01-20 18:33:55.136107: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344873 140191300101952 af.py:224] 2022-01-20 18:33:55.136112: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.344913 140191300101952 af.py:224] 2022-01-20 18:33:55.136126: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.344955 140191300101952 af.py:224] 2022-01-20 18:33:55.136134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.344996 140191300101952 af.py:224] 2022-01-20 18:33:55.136141: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345036 140191300101952 af.py:224] 2022-01-20 18:33:55.136145: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.345077 140191300101952 af.py:224] 2022-01-20 18:33:55.136151: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345117 140191300101952 af.py:224] 2022-01-20 18:33:55.136156: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.345158 140191300101952 af.py:224] 2022-01-20 18:33:55.136164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.345199 140191300101952 af.py:224] 2022-01-20 18:33:55.136170: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345256 140191300101952 af.py:224] 2022-01-20 18:33:55.136176: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345298 140191300101952 af.py:224] 2022-01-20 18:33:55.136182: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.345338 140191300101952 af.py:224] 2022-01-20 18:33:55.136187: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345379 140191300101952 af.py:224] 2022-01-20 18:33:55.136192: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.345420 140191300101952 af.py:224] 2022-01-20 18:33:55.136200: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.345460 140191300101952 af.py:224] 2022-01-20 18:33:55.136208: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345500 140191300101952 af.py:224] 2022-01-20 18:33:55.136214: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345541 140191300101952 af.py:224] 2022-01-20 18:33:55.136218: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.345581 140191300101952 af.py:224] 2022-01-20 18:33:55.136225: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345622 140191300101952 af.py:224] 2022-01-20 18:33:55.136230: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.345663 140191300101952 af.py:224] 2022-01-20 18:33:55.136238: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.345703 140191300101952 af.py:224] 2022-01-20 18:33:55.136244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345744 140191300101952 af.py:224] 2022-01-20 18:33:55.136250: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345785 140191300101952 af.py:224] 2022-01-20 18:33:55.136255: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.345825 140191300101952 af.py:224] 2022-01-20 18:33:55.136261: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345865 140191300101952 af.py:224] 2022-01-20 18:33:55.136266: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.345906 140191300101952 af.py:224] 2022-01-20 18:33:55.136274: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.345950 140191300101952 af.py:224] 2022-01-20 18:33:55.136280: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.345991 140191300101952 af.py:224] 2022-01-20 18:33:55.136287: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346032 140191300101952 af.py:224] 2022-01-20 18:33:55.136292: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.346071 140191300101952 af.py:224] 2022-01-20 18:33:55.136298: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346113 140191300101952 af.py:224] 2022-01-20 18:33:55.136304: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.346154 140191300101952 af.py:224] 2022-01-20 18:33:55.136312: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.346194 140191300101952 af.py:224] 2022-01-20 18:33:55.136319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346235 140191300101952 af.py:224] 2022-01-20 18:33:55.136325: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346275 140191300101952 af.py:224] 2022-01-20 18:33:55.136330: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.346316 140191300101952 af.py:224] 2022-01-20 18:33:55.136336: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346356 140191300101952 af.py:224] 2022-01-20 18:33:55.136341: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.346397 140191300101952 af.py:224] 2022-01-20 18:33:55.136348: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.346437 140191300101952 af.py:224] 2022-01-20 18:33:55.136355: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346477 140191300101952 af.py:224] 2022-01-20 18:33:55.136361: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346518 140191300101952 af.py:224] 2022-01-20 18:33:55.136366: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.346559 140191300101952 af.py:224] 2022-01-20 18:33:55.136372: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346599 140191300101952 af.py:224] 2022-01-20 18:33:55.136377: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.346643 140191300101952 af.py:224] 2022-01-20 18:33:55.136385: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.346684 140191300101952 af.py:224] 2022-01-20 18:33:55.136391: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346724 140191300101952 af.py:224] 2022-01-20 18:33:55.136397: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346765 140191300101952 af.py:224] 2022-01-20 18:33:55.136402: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.346805 140191300101952 af.py:224] 2022-01-20 18:33:55.136409: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.346863 140191300101952 af.py:224] 2022-01-20 18:33:55.136414: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.346905 140191300101952 af.py:224] 2022-01-20 18:33:55.136422: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.346971 140191300101952 af.py:224] 2022-01-20 18:33:55.136429: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347028 140191300101952 af.py:224] 2022-01-20 18:33:55.136436: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347069 140191300101952 af.py:224] 2022-01-20 18:33:55.136441: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.347127 140191300101952 af.py:224] 2022-01-20 18:33:55.136447: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347169 140191300101952 af.py:224] 2022-01-20 18:33:55.136452: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.347224 140191300101952 af.py:224] 2022-01-20 18:33:55.136461: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.347265 140191300101952 af.py:224] 2022-01-20 18:33:55.136467: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347306 140191300101952 af.py:224] 2022-01-20 18:33:55.136473: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347347 140191300101952 af.py:224] 2022-01-20 18:33:55.136478: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.347392 140191300101952 af.py:224] 2022-01-20 18:33:55.136484: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347433 140191300101952 af.py:224] 2022-01-20 18:33:55.136489: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.347473 140191300101952 af.py:224] 2022-01-20 18:33:55.136496: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.347514 140191300101952 af.py:224] 2022-01-20 18:33:55.136503: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347554 140191300101952 af.py:224] 2022-01-20 18:33:55.136509: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347595 140191300101952 af.py:224] 2022-01-20 18:33:55.136514: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.347635 140191300101952 af.py:224] 2022-01-20 18:33:55.136520: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347676 140191300101952 af.py:224] 2022-01-20 18:33:55.136525: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.347717 140191300101952 af.py:224] 2022-01-20 18:33:55.136534: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.347758 140191300101952 af.py:224] 2022-01-20 18:33:55.136540: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347798 140191300101952 af.py:224] 2022-01-20 18:33:55.136546: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347839 140191300101952 af.py:224] 2022-01-20 18:33:55.136551: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.347879 140191300101952 af.py:224] 2022-01-20 18:33:55.136558: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.347921 140191300101952 af.py:224] 2022-01-20 18:33:55.136563: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.347961 140191300101952 af.py:224] 2022-01-20 18:33:55.136571: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.348001 140191300101952 af.py:224] 2022-01-20 18:33:55.136578: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348042 140191300101952 af.py:224] 2022-01-20 18:33:55.136585: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348086 140191300101952 af.py:224] 2022-01-20 18:33:55.136590: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.348131 140191300101952 af.py:224] 2022-01-20 18:33:55.136596: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348173 140191300101952 af.py:224] 2022-01-20 18:33:55.136601: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.348213 140191300101952 af.py:224] 2022-01-20 18:33:55.136609: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.348254 140191300101952 af.py:224] 2022-01-20 18:33:55.136615: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348295 140191300101952 af.py:224] 2022-01-20 18:33:55.136621: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348336 140191300101952 af.py:224] 2022-01-20 18:33:55.136626: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.348376 140191300101952 af.py:224] 2022-01-20 18:33:55.136632: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348417 140191300101952 af.py:224] 2022-01-20 18:33:55.136637: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.348458 140191300101952 af.py:224] 2022-01-20 18:33:55.136646: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.348499 140191300101952 af.py:224] 2022-01-20 18:33:55.136653: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348552 140191300101952 af.py:224] 2022-01-20 18:33:55.136659: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348594 140191300101952 af.py:224] 2022-01-20 18:33:55.136664: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.348634 140191300101952 af.py:224] 2022-01-20 18:33:55.136671: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348675 140191300101952 af.py:224] 2022-01-20 18:33:55.136676: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.348716 140191300101952 af.py:224] 2022-01-20 18:33:55.136684: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.348757 140191300101952 af.py:224] 2022-01-20 18:33:55.136690: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348801 140191300101952 af.py:224] 2022-01-20 18:33:55.136697: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348843 140191300101952 af.py:224] 2022-01-20 18:33:55.136702: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.348883 140191300101952 af.py:224] 2022-01-20 18:33:55.136707: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.348924 140191300101952 af.py:224] 2022-01-20 18:33:55.136712: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.348964 140191300101952 af.py:224] 2022-01-20 18:33:55.136720: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.349004 140191300101952 af.py:224] 2022-01-20 18:33:55.136727: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349045 140191300101952 af.py:224] 2022-01-20 18:33:55.136733: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349086 140191300101952 af.py:224] 2022-01-20 18:33:55.136738: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.349128 140191300101952 af.py:224] 2022-01-20 18:33:55.136743: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349169 140191300101952 af.py:224] 2022-01-20 18:33:55.136748: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.349210 140191300101952 af.py:224] 2022-01-20 18:33:55.136756: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.349251 140191300101952 af.py:224] 2022-01-20 18:33:55.136762: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349292 140191300101952 af.py:224] 2022-01-20 18:33:55.136768: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349333 140191300101952 af.py:224] 2022-01-20 18:33:55.136773: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.349373 140191300101952 af.py:224] 2022-01-20 18:33:55.136778: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349414 140191300101952 af.py:224] 2022-01-20 18:33:55.136783: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.349454 140191300101952 af.py:224] 2022-01-20 18:33:55.136790: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.349498 140191300101952 af.py:224] 2022-01-20 18:33:55.136797: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349539 140191300101952 af.py:224] 2022-01-20 18:33:55.136803: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349580 140191300101952 af.py:224] 2022-01-20 18:33:55.136808: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.349621 140191300101952 af.py:224] 2022-01-20 18:33:55.136815: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349661 140191300101952 af.py:224] 2022-01-20 18:33:55.136819: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.349701 140191300101952 af.py:224] 2022-01-20 18:33:55.136827: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.349742 140191300101952 af.py:224] 2022-01-20 18:33:55.136834: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349782 140191300101952 af.py:224] 2022-01-20 18:33:55.136840: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349823 140191300101952 af.py:224] 2022-01-20 18:33:55.136844: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.349864 140191300101952 af.py:224] 2022-01-20 18:33:55.136850: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.349905 140191300101952 af.py:224] 2022-01-20 18:33:55.136854: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.349945 140191300101952 af.py:224] 2022-01-20 18:33:55.136863: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.349986 140191300101952 af.py:224] 2022-01-20 18:33:55.136869: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350026 140191300101952 af.py:224] 2022-01-20 18:33:55.136875: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350066 140191300101952 af.py:224] 2022-01-20 18:33:55.136879: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.350107 140191300101952 af.py:224] 2022-01-20 18:33:55.136886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350151 140191300101952 af.py:224] 2022-01-20 18:33:55.136891: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.350207 140191300101952 af.py:224] 2022-01-20 18:33:55.136898: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.350249 140191300101952 af.py:224] 2022-01-20 18:33:55.136904: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350290 140191300101952 af.py:224] 2022-01-20 18:33:55.136910: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350331 140191300101952 af.py:224] 2022-01-20 18:33:55.136914: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.350371 140191300101952 af.py:224] 2022-01-20 18:33:55.136920: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350413 140191300101952 af.py:224] 2022-01-20 18:33:55.136925: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.350453 140191300101952 af.py:224] 2022-01-20 18:33:55.136933: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.350494 140191300101952 af.py:224] 2022-01-20 18:33:55.136940: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350534 140191300101952 af.py:224] 2022-01-20 18:33:55.136946: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350574 140191300101952 af.py:224] 2022-01-20 18:33:55.136952: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.350615 140191300101952 af.py:224] 2022-01-20 18:33:55.136958: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350655 140191300101952 af.py:224] 2022-01-20 18:33:55.136963: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.350696 140191300101952 af.py:224] 2022-01-20 18:33:55.136971: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.350737 140191300101952 af.py:224] 2022-01-20 18:33:55.136978: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350778 140191300101952 af.py:224] 2022-01-20 18:33:55.136984: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350819 140191300101952 af.py:224] 2022-01-20 18:33:55.136989: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.350862 140191300101952 af.py:224] 2022-01-20 18:33:55.136995: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.350903 140191300101952 af.py:224] 2022-01-20 18:33:55.137000: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.350944 140191300101952 af.py:224] 2022-01-20 18:33:55.137008: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.350986 140191300101952 af.py:224] 2022-01-20 18:33:55.137015: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351027 140191300101952 af.py:224] 2022-01-20 18:33:55.137021: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351068 140191300101952 af.py:224] 2022-01-20 18:33:55.137026: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.351108 140191300101952 af.py:224] 2022-01-20 18:33:55.137033: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351149 140191300101952 af.py:224] 2022-01-20 18:33:55.137038: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.351190 140191300101952 af.py:224] 2022-01-20 18:33:55.137045: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.351230 140191300101952 af.py:224] 2022-01-20 18:33:55.137052: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351271 140191300101952 af.py:224] 2022-01-20 18:33:55.137058: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351312 140191300101952 af.py:224] 2022-01-20 18:33:55.137063: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.351352 140191300101952 af.py:224] 2022-01-20 18:33:55.137069: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351393 140191300101952 af.py:224] 2022-01-20 18:33:55.137074: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.351433 140191300101952 af.py:224] 2022-01-20 18:33:55.137082: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.351474 140191300101952 af.py:224] 2022-01-20 18:33:55.137088: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351515 140191300101952 af.py:224] 2022-01-20 18:33:55.137094: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351559 140191300101952 af.py:224] 2022-01-20 18:33:55.137099: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.351600 140191300101952 af.py:224] 2022-01-20 18:33:55.137105: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351640 140191300101952 af.py:224] 2022-01-20 18:33:55.137110: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.351680 140191300101952 af.py:224] 2022-01-20 18:33:55.137119: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.351721 140191300101952 af.py:224] 2022-01-20 18:33:55.137126: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351761 140191300101952 af.py:224] 2022-01-20 18:33:55.137132: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351817 140191300101952 af.py:224] 2022-01-20 18:33:55.137137: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.351859 140191300101952 af.py:224] 2022-01-20 18:33:55.137143: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.351900 140191300101952 af.py:224] 2022-01-20 18:33:55.137149: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.351940 140191300101952 af.py:224] 2022-01-20 18:33:55.137157: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.351981 140191300101952 af.py:224] 2022-01-20 18:33:55.137164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352021 140191300101952 af.py:224] 2022-01-20 18:33:55.137170: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352061 140191300101952 af.py:224] 2022-01-20 18:33:55.137175: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.352101 140191300101952 af.py:224] 2022-01-20 18:33:55.137181: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352146 140191300101952 af.py:224] 2022-01-20 18:33:55.137186: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.352187 140191300101952 af.py:224] 2022-01-20 18:33:55.137194: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.352227 140191300101952 af.py:224] 2022-01-20 18:33:55.137201: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352271 140191300101952 af.py:224] 2022-01-20 18:33:55.137207: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352312 140191300101952 af.py:224] 2022-01-20 18:33:55.137212: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.352353 140191300101952 af.py:224] 2022-01-20 18:33:55.137218: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352393 140191300101952 af.py:224] 2022-01-20 18:33:55.137223: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.352434 140191300101952 af.py:224] 2022-01-20 18:33:55.137231: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.352475 140191300101952 af.py:224] 2022-01-20 18:33:55.137237: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352515 140191300101952 af.py:224] 2022-01-20 18:33:55.137244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352556 140191300101952 af.py:224] 2022-01-20 18:33:55.137249: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.352596 140191300101952 af.py:224] 2022-01-20 18:33:55.137254: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352638 140191300101952 af.py:224] 2022-01-20 18:33:55.137259: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.352678 140191300101952 af.py:224] 2022-01-20 18:33:55.137267: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.352719 140191300101952 af.py:224] 2022-01-20 18:33:55.137274: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352759 140191300101952 af.py:224] 2022-01-20 18:33:55.137280: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352800 140191300101952 af.py:224] 2022-01-20 18:33:55.137285: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.352840 140191300101952 af.py:224] 2022-01-20 18:33:55.137291: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.352881 140191300101952 af.py:224] 2022-01-20 18:33:55.137296: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.352921 140191300101952 af.py:224] 2022-01-20 18:33:55.137305: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.352964 140191300101952 af.py:224] 2022-01-20 18:33:55.137313: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353005 140191300101952 af.py:224] 2022-01-20 18:33:55.137319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353046 140191300101952 af.py:224] 2022-01-20 18:33:55.137323: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.353087 140191300101952 af.py:224] 2022-01-20 18:33:55.137330: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353127 140191300101952 af.py:224] 2022-01-20 18:33:55.137335: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.353168 140191300101952 af.py:224] 2022-01-20 18:33:55.137342: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.353209 140191300101952 af.py:224] 2022-01-20 18:33:55.137349: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353249 140191300101952 af.py:224] 2022-01-20 18:33:55.137355: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353290 140191300101952 af.py:224] 2022-01-20 18:33:55.137359: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.353330 140191300101952 af.py:224] 2022-01-20 18:33:55.137366: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353371 140191300101952 af.py:224] 2022-01-20 18:33:55.137371: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.353429 140191300101952 af.py:224] 2022-01-20 18:33:55.137378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.353471 140191300101952 af.py:224] 2022-01-20 18:33:55.137385: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353545 140191300101952 af.py:224] 2022-01-20 18:33:55.137391: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353609 140191300101952 af.py:224] 2022-01-20 18:33:55.137396: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.353663 140191300101952 af.py:224] 2022-01-20 18:33:55.137402: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353711 140191300101952 af.py:224] 2022-01-20 18:33:55.137407: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.353752 140191300101952 af.py:224] 2022-01-20 18:33:55.137416: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.353808 140191300101952 af.py:224] 2022-01-20 18:33:55.137423: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353848 140191300101952 af.py:224] 2022-01-20 18:33:55.137429: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353889 140191300101952 af.py:224] 2022-01-20 18:33:55.137434: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.353930 140191300101952 af.py:224] 2022-01-20 18:33:55.137441: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.353971 140191300101952 af.py:224] 2022-01-20 18:33:55.137446: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.354012 140191300101952 af.py:224] 2022-01-20 18:33:55.137454: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.354053 140191300101952 af.py:224] 2022-01-20 18:33:55.137460: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354094 140191300101952 af.py:224] 2022-01-20 18:33:55.137467: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354135 140191300101952 af.py:224] 2022-01-20 18:33:55.137472: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.354175 140191300101952 af.py:224] 2022-01-20 18:33:55.137478: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354215 140191300101952 af.py:224] 2022-01-20 18:33:55.137483: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.354256 140191300101952 af.py:224] 2022-01-20 18:33:55.137490: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.354297 140191300101952 af.py:224] 2022-01-20 18:33:55.137498: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354337 140191300101952 af.py:224] 2022-01-20 18:33:55.137504: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354377 140191300101952 af.py:224] 2022-01-20 18:33:55.137508: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.354423 140191300101952 af.py:224] 2022-01-20 18:33:55.137514: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354464 140191300101952 af.py:224] 2022-01-20 18:33:55.137519: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.354504 140191300101952 af.py:224] 2022-01-20 18:33:55.137527: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.354545 140191300101952 af.py:224] 2022-01-20 18:33:55.137534: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354586 140191300101952 af.py:224] 2022-01-20 18:33:55.137540: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354626 140191300101952 af.py:224] 2022-01-20 18:33:55.137545: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.354666 140191300101952 af.py:224] 2022-01-20 18:33:55.137550: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354707 140191300101952 af.py:224] 2022-01-20 18:33:55.137555: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.354747 140191300101952 af.py:224] 2022-01-20 18:33:55.137564: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.354788 140191300101952 af.py:224] 2022-01-20 18:33:55.137570: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354828 140191300101952 af.py:224] 2022-01-20 18:33:55.137576: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354869 140191300101952 af.py:224] 2022-01-20 18:33:55.137581: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.354909 140191300101952 af.py:224] 2022-01-20 18:33:55.137587: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.354950 140191300101952 af.py:224] 2022-01-20 18:33:55.137592: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.354991 140191300101952 af.py:224] 2022-01-20 18:33:55.137600: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.355031 140191300101952 af.py:224] 2022-01-20 18:33:55.137606: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355071 140191300101952 af.py:224] 2022-01-20 18:33:55.137612: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355134 140191300101952 af.py:224] 2022-01-20 18:33:55.137617: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.355175 140191300101952 af.py:224] 2022-01-20 18:33:55.137624: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355215 140191300101952 af.py:224] 2022-01-20 18:33:55.137629: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.355256 140191300101952 af.py:224] 2022-01-20 18:33:55.137636: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.355296 140191300101952 af.py:224] 2022-01-20 18:33:55.137643: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355337 140191300101952 af.py:224] 2022-01-20 18:33:55.137650: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355378 140191300101952 af.py:224] 2022-01-20 18:33:55.137654: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.355419 140191300101952 af.py:224] 2022-01-20 18:33:55.137660: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355460 140191300101952 af.py:224] 2022-01-20 18:33:55.137665: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.355501 140191300101952 af.py:224] 2022-01-20 18:33:55.137673: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.355541 140191300101952 af.py:224] 2022-01-20 18:33:55.137680: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355582 140191300101952 af.py:224] 2022-01-20 18:33:55.137686: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355623 140191300101952 af.py:224] 2022-01-20 18:33:55.137691: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.355664 140191300101952 af.py:224] 2022-01-20 18:33:55.137697: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355705 140191300101952 af.py:224] 2022-01-20 18:33:55.137702: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.355746 140191300101952 af.py:224] 2022-01-20 18:33:55.137710: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.355787 140191300101952 af.py:224] 2022-01-20 18:33:55.137717: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355831 140191300101952 af.py:224] 2022-01-20 18:33:55.137722: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355872 140191300101952 af.py:224] 2022-01-20 18:33:55.137726: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.355913 140191300101952 af.py:224] 2022-01-20 18:33:55.137732: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.355953 140191300101952 af.py:224] 2022-01-20 18:33:55.137737: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.355994 140191300101952 af.py:224] 2022-01-20 18:33:55.137745: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.356034 140191300101952 af.py:224] 2022-01-20 18:33:55.137751: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356075 140191300101952 af.py:224] 2022-01-20 18:33:55.137757: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356120 140191300101952 af.py:224] 2022-01-20 18:33:55.137761: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.356217 140191300101952 af.py:224] 2022-01-20 18:33:55.137768: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356258 140191300101952 af.py:224] 2022-01-20 18:33:55.137772: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.356299 140191300101952 af.py:224] 2022-01-20 18:33:55.137780: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.356339 140191300101952 af.py:224] 2022-01-20 18:33:55.137787: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356380 140191300101952 af.py:224] 2022-01-20 18:33:55.137793: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356421 140191300101952 af.py:224] 2022-01-20 18:33:55.137797: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.356461 140191300101952 af.py:224] 2022-01-20 18:33:55.137803: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356502 140191300101952 af.py:224] 2022-01-20 18:33:55.137808: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.356542 140191300101952 af.py:224] 2022-01-20 18:33:55.137816: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.356587 140191300101952 af.py:224] 2022-01-20 18:33:55.137823: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356628 140191300101952 af.py:224] 2022-01-20 18:33:55.137829: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356669 140191300101952 af.py:224] 2022-01-20 18:33:55.137835: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.356709 140191300101952 af.py:224] 2022-01-20 18:33:55.137841: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356750 140191300101952 af.py:224] 2022-01-20 18:33:55.137846: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.356790 140191300101952 af.py:224] 2022-01-20 18:33:55.137854: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.356845 140191300101952 af.py:224] 2022-01-20 18:33:55.137861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356887 140191300101952 af.py:224] 2022-01-20 18:33:55.137867: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.356928 140191300101952 af.py:224] 2022-01-20 18:33:55.137871: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.356969 140191300101952 af.py:224] 2022-01-20 18:33:55.137877: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357010 140191300101952 af.py:224] 2022-01-20 18:33:55.137882: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.357051 140191300101952 af.py:224] 2022-01-20 18:33:55.137890: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.357091 140191300101952 af.py:224] 2022-01-20 18:33:55.137897: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357132 140191300101952 af.py:224] 2022-01-20 18:33:55.137903: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357173 140191300101952 af.py:224] 2022-01-20 18:33:55.137909: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.357214 140191300101952 af.py:224] 2022-01-20 18:33:55.137915: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357259 140191300101952 af.py:224] 2022-01-20 18:33:55.137920: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.357300 140191300101952 af.py:224] 2022-01-20 18:33:55.137929: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.357341 140191300101952 af.py:224] 2022-01-20 18:33:55.137936: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357383 140191300101952 af.py:224] 2022-01-20 18:33:55.137942: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357423 140191300101952 af.py:224] 2022-01-20 18:33:55.137947: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.357464 140191300101952 af.py:224] 2022-01-20 18:33:55.137953: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357505 140191300101952 af.py:224] 2022-01-20 18:33:55.137959: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.357546 140191300101952 af.py:224] 2022-01-20 18:33:55.137967: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.357587 140191300101952 af.py:224] 2022-01-20 18:33:55.137973: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357628 140191300101952 af.py:224] 2022-01-20 18:33:55.137980: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357669 140191300101952 af.py:224] 2022-01-20 18:33:55.137985: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.357710 140191300101952 af.py:224] 2022-01-20 18:33:55.137992: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357751 140191300101952 af.py:224] 2022-01-20 18:33:55.137997: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.357791 140191300101952 af.py:224] 2022-01-20 18:33:55.138006: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.357832 140191300101952 af.py:224] 2022-01-20 18:33:55.138012: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357873 140191300101952 af.py:224] 2022-01-20 18:33:55.138019: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357914 140191300101952 af.py:224] 2022-01-20 18:33:55.138024: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.357958 140191300101952 af.py:224] 2022-01-20 18:33:55.138030: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.357999 140191300101952 af.py:224] 2022-01-20 18:33:55.138036: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.358039 140191300101952 af.py:224] 2022-01-20 18:33:55.138043: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.358080 140191300101952 af.py:224] 2022-01-20 18:33:55.138051: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358121 140191300101952 af.py:224] 2022-01-20 18:33:55.138057: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358162 140191300101952 af.py:224] 2022-01-20 18:33:55.138062: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.358203 140191300101952 af.py:224] 2022-01-20 18:33:55.138069: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358243 140191300101952 af.py:224] 2022-01-20 18:33:55.138074: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.358284 140191300101952 af.py:224] 2022-01-20 18:33:55.138083: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.358324 140191300101952 af.py:224] 2022-01-20 18:33:55.138089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358365 140191300101952 af.py:224] 2022-01-20 18:33:55.138096: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358405 140191300101952 af.py:224] 2022-01-20 18:33:55.138101: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.358461 140191300101952 af.py:224] 2022-01-20 18:33:55.138108: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358503 140191300101952 af.py:224] 2022-01-20 18:33:55.138113: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.358544 140191300101952 af.py:224] 2022-01-20 18:33:55.138121: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.358584 140191300101952 af.py:224] 2022-01-20 18:33:55.138128: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358625 140191300101952 af.py:224] 2022-01-20 18:33:55.138134: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358669 140191300101952 af.py:224] 2022-01-20 18:33:55.138139: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.358710 140191300101952 af.py:224] 2022-01-20 18:33:55.138145: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358751 140191300101952 af.py:224] 2022-01-20 18:33:55.138150: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.358791 140191300101952 af.py:224] 2022-01-20 18:33:55.138160: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.358832 140191300101952 af.py:224] 2022-01-20 18:33:55.138167: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358872 140191300101952 af.py:224] 2022-01-20 18:33:55.138174: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358913 140191300101952 af.py:224] 2022-01-20 18:33:55.138179: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.358954 140191300101952 af.py:224] 2022-01-20 18:33:55.138186: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.358994 140191300101952 af.py:224] 2022-01-20 18:33:55.138191: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.359035 140191300101952 af.py:224] 2022-01-20 18:33:55.138198: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.359076 140191300101952 af.py:224] 2022-01-20 18:33:55.138205: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359117 140191300101952 af.py:224] 2022-01-20 18:33:55.138211: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359157 140191300101952 af.py:224] 2022-01-20 18:33:55.138216: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.359198 140191300101952 af.py:224] 2022-01-20 18:33:55.138223: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359239 140191300101952 af.py:224] 2022-01-20 18:33:55.138228: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.359280 140191300101952 af.py:224] 2022-01-20 18:33:55.138236: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.359321 140191300101952 af.py:224] 2022-01-20 18:33:55.138242: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359364 140191300101952 af.py:224] 2022-01-20 18:33:55.138248: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359405 140191300101952 af.py:224] 2022-01-20 18:33:55.138253: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.359446 140191300101952 af.py:224] 2022-01-20 18:33:55.138260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359487 140191300101952 af.py:224] 2022-01-20 18:33:55.138265: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.359527 140191300101952 af.py:224] 2022-01-20 18:33:55.138273: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.359568 140191300101952 af.py:224] 2022-01-20 18:33:55.138280: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359609 140191300101952 af.py:224] 2022-01-20 18:33:55.138286: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359650 140191300101952 af.py:224] 2022-01-20 18:33:55.138291: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.359690 140191300101952 af.py:224] 2022-01-20 18:33:55.138298: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359731 140191300101952 af.py:224] 2022-01-20 18:33:55.138303: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.359772 140191300101952 af.py:224] 2022-01-20 18:33:55.138312: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.359813 140191300101952 af.py:224] 2022-01-20 18:33:55.138319: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359853 140191300101952 af.py:224] 2022-01-20 18:33:55.138325: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359894 140191300101952 af.py:224] 2022-01-20 18:33:55.138330: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.359935 140191300101952 af.py:224] 2022-01-20 18:33:55.138336: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.359976 140191300101952 af.py:224] 2022-01-20 18:33:55.138341: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.360017 140191300101952 af.py:224] 2022-01-20 18:33:55.138348: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.360079 140191300101952 af.py:224] 2022-01-20 18:33:55.138355: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.360128 140191300101952 af.py:224] 2022-01-20 18:33:55.138361: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.360217 140191300101952 af.py:224] 2022-01-20 18:33:55.138366: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.360327 140191300101952 af.py:224] 2022-01-20 18:33:55.138373: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.360385 140191300101952 af.py:224] 2022-01-20 18:33:55.138378: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.360441 140191300101952 af.py:224] 2022-01-20 18:33:55.138387: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.360482 140191300101952 af.py:224] 2022-01-20 18:33:55.138393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.360523 140191300101952 af.py:224] 2022-01-20 18:33:55.138399: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.360564 140191300101952 af.py:224] 2022-01-20 18:33:55.138404: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.360605 140191300101952 af.py:224] 2022-01-20 18:33:55.138411: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.360646 140191300101952 af.py:224] 2022-01-20 18:33:55.138416: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.360687 140191300101952 af.py:224] 2022-01-20 18:33:55.138424: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.360728 140191300101952 af.py:224] 2022-01-20 18:33:55.138431: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.360768 140191300101952 af.py:224] 2022-01-20 18:33:55.138437: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.360809 140191300101952 af.py:224] 2022-01-20 18:33:55.138442: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.360849 140191300101952 af.py:224] 2022-01-20 18:33:55.138449: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.360893 140191300101952 af.py:224] 2022-01-20 18:33:55.138454: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.360935 140191300101952 af.py:224] 2022-01-20 18:33:55.138463: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.360975 140191300101952 af.py:224] 2022-01-20 18:33:55.138470: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361016 140191300101952 af.py:224] 2022-01-20 18:33:55.138476: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361056 140191300101952 af.py:224] 2022-01-20 18:33:55.138481: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.361096 140191300101952 af.py:224] 2022-01-20 18:33:55.138487: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361137 140191300101952 af.py:224] 2022-01-20 18:33:55.138492: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.361178 140191300101952 af.py:224] 2022-01-20 18:33:55.138500: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.361219 140191300101952 af.py:224] 2022-01-20 18:33:55.138507: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361259 140191300101952 af.py:224] 2022-01-20 18:33:55.138513: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361300 140191300101952 af.py:224] 2022-01-20 18:33:55.138518: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.361341 140191300101952 af.py:224] 2022-01-20 18:33:55.138525: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361382 140191300101952 af.py:224] 2022-01-20 18:33:55.138530: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.361423 140191300101952 af.py:224] 2022-01-20 18:33:55.138538: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.361464 140191300101952 af.py:224] 2022-01-20 18:33:55.138545: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361505 140191300101952 af.py:224] 2022-01-20 18:33:55.138551: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361546 140191300101952 af.py:224] 2022-01-20 18:33:55.138556: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.361589 140191300101952 af.py:224] 2022-01-20 18:33:55.138563: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361630 140191300101952 af.py:224] 2022-01-20 18:33:55.138568: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.361671 140191300101952 af.py:224] 2022-01-20 18:33:55.138575: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.361713 140191300101952 af.py:224] 2022-01-20 18:33:55.138583: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361753 140191300101952 af.py:224] 2022-01-20 18:33:55.138589: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361794 140191300101952 af.py:224] 2022-01-20 18:33:55.138594: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.361851 140191300101952 af.py:224] 2022-01-20 18:33:55.138601: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.361893 140191300101952 af.py:224] 2022-01-20 18:33:55.138606: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.361933 140191300101952 af.py:224] 2022-01-20 18:33:55.138614: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.361974 140191300101952 af.py:224] 2022-01-20 18:33:55.138621: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362014 140191300101952 af.py:224] 2022-01-20 18:33:55.138627: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362055 140191300101952 af.py:224] 2022-01-20 18:33:55.138632: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.362096 140191300101952 af.py:224] 2022-01-20 18:33:55.138638: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362137 140191300101952 af.py:224] 2022-01-20 18:33:55.138643: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.362177 140191300101952 af.py:224] 2022-01-20 18:33:55.138651: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.362217 140191300101952 af.py:224] 2022-01-20 18:33:55.138658: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362258 140191300101952 af.py:224] 2022-01-20 18:33:55.138664: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362303 140191300101952 af.py:224] 2022-01-20 18:33:55.138669: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.362344 140191300101952 af.py:224] 2022-01-20 18:33:55.138676: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362385 140191300101952 af.py:224] 2022-01-20 18:33:55.138681: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.362426 140191300101952 af.py:224] 2022-01-20 18:33:55.138689: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.362466 140191300101952 af.py:224] 2022-01-20 18:33:55.138696: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362507 140191300101952 af.py:224] 2022-01-20 18:33:55.138703: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362548 140191300101952 af.py:224] 2022-01-20 18:33:55.138708: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.362588 140191300101952 af.py:224] 2022-01-20 18:33:55.138714: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362628 140191300101952 af.py:224] 2022-01-20 18:33:55.138719: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.362668 140191300101952 af.py:224] 2022-01-20 18:33:55.138727: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.362709 140191300101952 af.py:224] 2022-01-20 18:33:55.138734: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362750 140191300101952 af.py:224] 2022-01-20 18:33:55.138740: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362790 140191300101952 af.py:224] 2022-01-20 18:33:55.138745: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.362831 140191300101952 af.py:224] 2022-01-20 18:33:55.138752: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362872 140191300101952 af.py:224] 2022-01-20 18:33:55.138757: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.362912 140191300101952 af.py:224] 2022-01-20 18:33:55.138765: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.362953 140191300101952 af.py:224] 2022-01-20 18:33:55.138772: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.362997 140191300101952 af.py:224] 2022-01-20 18:33:55.138778: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363039 140191300101952 af.py:224] 2022-01-20 18:33:55.138783: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.363079 140191300101952 af.py:224] 2022-01-20 18:33:55.138789: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363121 140191300101952 af.py:224] 2022-01-20 18:33:55.138794: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.363161 140191300101952 af.py:224] 2022-01-20 18:33:55.138802: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.363202 140191300101952 af.py:224] 2022-01-20 18:33:55.138810: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363243 140191300101952 af.py:224] 2022-01-20 18:33:55.138817: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363283 140191300101952 af.py:224] 2022-01-20 18:33:55.138822: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.363324 140191300101952 af.py:224] 2022-01-20 18:33:55.138829: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363365 140191300101952 af.py:224] 2022-01-20 18:33:55.138834: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.363405 140191300101952 af.py:224] 2022-01-20 18:33:55.138842: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.363459 140191300101952 af.py:224] 2022-01-20 18:33:55.138849: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363500 140191300101952 af.py:224] 2022-01-20 18:33:55.138855: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363541 140191300101952 af.py:224] 2022-01-20 18:33:55.138860: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.363581 140191300101952 af.py:224] 2022-01-20 18:33:55.138867: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363622 140191300101952 af.py:224] 2022-01-20 18:33:55.138872: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.363662 140191300101952 af.py:224] 2022-01-20 18:33:55.138880: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.363706 140191300101952 af.py:224] 2022-01-20 18:33:55.138887: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363747 140191300101952 af.py:224] 2022-01-20 18:33:55.138893: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363788 140191300101952 af.py:224] 2022-01-20 18:33:55.138898: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.363828 140191300101952 af.py:224] 2022-01-20 18:33:55.138905: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363869 140191300101952 af.py:224] 2022-01-20 18:33:55.138910: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.363909 140191300101952 af.py:224] 2022-01-20 18:33:55.138918: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.363950 140191300101952 af.py:224] 2022-01-20 18:33:55.138924: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.363991 140191300101952 af.py:224] 2022-01-20 18:33:55.138930: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364032 140191300101952 af.py:224] 2022-01-20 18:33:55.138936: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.364072 140191300101952 af.py:224] 2022-01-20 18:33:55.138942: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364113 140191300101952 af.py:224] 2022-01-20 18:33:55.138947: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.364215 140191300101952 af.py:224] 2022-01-20 18:33:55.138955: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.364255 140191300101952 af.py:224] 2022-01-20 18:33:55.138961: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364296 140191300101952 af.py:224] 2022-01-20 18:33:55.138967: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364336 140191300101952 af.py:224] 2022-01-20 18:33:55.138972: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.364377 140191300101952 af.py:224] 2022-01-20 18:33:55.138979: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364422 140191300101952 af.py:224] 2022-01-20 18:33:55.138984: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.364463 140191300101952 af.py:224] 2022-01-20 18:33:55.138992: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.364503 140191300101952 af.py:224] 2022-01-20 18:33:55.138999: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364543 140191300101952 af.py:224] 2022-01-20 18:33:55.139006: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364585 140191300101952 af.py:224] 2022-01-20 18:33:55.139011: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.364626 140191300101952 af.py:224] 2022-01-20 18:33:55.139017: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364668 140191300101952 af.py:224] 2022-01-20 18:33:55.139022: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.364708 140191300101952 af.py:224] 2022-01-20 18:33:55.139030: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.364749 140191300101952 af.py:224] 2022-01-20 18:33:55.139037: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364790 140191300101952 af.py:224] 2022-01-20 18:33:55.139043: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364831 140191300101952 af.py:224] 2022-01-20 18:33:55.139048: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.364872 140191300101952 af.py:224] 2022-01-20 18:33:55.139054: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.364912 140191300101952 af.py:224] 2022-01-20 18:33:55.139059: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.364953 140191300101952 af.py:224] 2022-01-20 18:33:55.139067: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.364994 140191300101952 af.py:224] 2022-01-20 18:33:55.139073: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365034 140191300101952 af.py:224] 2022-01-20 18:33:55.139080: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365075 140191300101952 af.py:224] 2022-01-20 18:33:55.139085: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.365135 140191300101952 af.py:224] 2022-01-20 18:33:55.139092: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365178 140191300101952 af.py:224] 2022-01-20 18:33:55.139097: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.365219 140191300101952 af.py:224] 2022-01-20 18:33:55.139104: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.365260 140191300101952 af.py:224] 2022-01-20 18:33:55.139111: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365300 140191300101952 af.py:224] 2022-01-20 18:33:55.139117: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365341 140191300101952 af.py:224] 2022-01-20 18:33:55.139122: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.365381 140191300101952 af.py:224] 2022-01-20 18:33:55.139129: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365422 140191300101952 af.py:224] 2022-01-20 18:33:55.139134: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.365462 140191300101952 af.py:224] 2022-01-20 18:33:55.139142: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.365503 140191300101952 af.py:224] 2022-01-20 18:33:55.139149: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365544 140191300101952 af.py:224] 2022-01-20 18:33:55.139155: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365585 140191300101952 af.py:224] 2022-01-20 18:33:55.139160: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.365626 140191300101952 af.py:224] 2022-01-20 18:33:55.139167: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365667 140191300101952 af.py:224] 2022-01-20 18:33:55.139172: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.365707 140191300101952 af.py:224] 2022-01-20 18:33:55.139180: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.365747 140191300101952 af.py:224] 2022-01-20 18:33:55.139187: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365788 140191300101952 af.py:224] 2022-01-20 18:33:55.139194: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365834 140191300101952 af.py:224] 2022-01-20 18:33:55.139199: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.365875 140191300101952 af.py:224] 2022-01-20 18:33:55.139206: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.365917 140191300101952 af.py:224] 2022-01-20 18:33:55.139211: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.365957 140191300101952 af.py:224] 2022-01-20 18:33:55.139220: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.365998 140191300101952 af.py:224] 2022-01-20 18:33:55.139226: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366038 140191300101952 af.py:224] 2022-01-20 18:33:55.139233: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366079 140191300101952 af.py:224] 2022-01-20 18:33:55.139237: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.366120 140191300101952 af.py:224] 2022-01-20 18:33:55.139244: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366161 140191300101952 af.py:224] 2022-01-20 18:33:55.139249: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.366201 140191300101952 af.py:224] 2022-01-20 18:33:55.139257: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.366242 140191300101952 af.py:224] 2022-01-20 18:33:55.139263: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366282 140191300101952 af.py:224] 2022-01-20 18:33:55.139270: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366323 140191300101952 af.py:224] 2022-01-20 18:33:55.139275: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.366364 140191300101952 af.py:224] 2022-01-20 18:33:55.139282: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366404 140191300101952 af.py:224] 2022-01-20 18:33:55.139287: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.366445 140191300101952 af.py:224] 2022-01-20 18:33:55.139295: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.366485 140191300101952 af.py:224] 2022-01-20 18:33:55.139302: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366529 140191300101952 af.py:224] 2022-01-20 18:33:55.139308: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366570 140191300101952 af.py:224] 2022-01-20 18:33:55.139313: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.366611 140191300101952 af.py:224] 2022-01-20 18:33:55.139320: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366652 140191300101952 af.py:224] 2022-01-20 18:33:55.139325: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.366692 140191300101952 af.py:224] 2022-01-20 18:33:55.139333: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.366733 140191300101952 af.py:224] 2022-01-20 18:33:55.139340: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366791 140191300101952 af.py:224] 2022-01-20 18:33:55.139346: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366833 140191300101952 af.py:224] 2022-01-20 18:33:55.139351: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.366874 140191300101952 af.py:224] 2022-01-20 18:33:55.139358: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.366931 140191300101952 af.py:224] 2022-01-20 18:33:55.139363: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.366987 140191300101952 af.py:224] 2022-01-20 18:33:55.139371: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.367028 140191300101952 af.py:224] 2022-01-20 18:33:55.139378: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367069 140191300101952 af.py:224] 2022-01-20 18:33:55.139384: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367109 140191300101952 af.py:224] 2022-01-20 18:33:55.139389: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.367150 140191300101952 af.py:224] 2022-01-20 18:33:55.139396: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367191 140191300101952 af.py:224] 2022-01-20 18:33:55.139401: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.367232 140191300101952 af.py:224] 2022-01-20 18:33:55.139409: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.367275 140191300101952 af.py:224] 2022-01-20 18:33:55.139415: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367317 140191300101952 af.py:224] 2022-01-20 18:33:55.139421: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367358 140191300101952 af.py:224] 2022-01-20 18:33:55.139426: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.367399 140191300101952 af.py:224] 2022-01-20 18:33:55.139433: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367440 140191300101952 af.py:224] 2022-01-20 18:33:55.139438: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.367481 140191300101952 af.py:224] 2022-01-20 18:33:55.139446: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.367522 140191300101952 af.py:224] 2022-01-20 18:33:55.139453: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367563 140191300101952 af.py:224] 2022-01-20 18:33:55.139459: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367604 140191300101952 af.py:224] 2022-01-20 18:33:55.139464: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.367644 140191300101952 af.py:224] 2022-01-20 18:33:55.139471: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367685 140191300101952 af.py:224] 2022-01-20 18:33:55.139476: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.367726 140191300101952 af.py:224] 2022-01-20 18:33:55.139484: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.367767 140191300101952 af.py:224] 2022-01-20 18:33:55.139491: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367808 140191300101952 af.py:224] 2022-01-20 18:33:55.139497: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367849 140191300101952 af.py:224] 2022-01-20 18:33:55.139502: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.367889 140191300101952 af.py:224] 2022-01-20 18:33:55.139508: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.367933 140191300101952 af.py:224] 2022-01-20 18:33:55.139513: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.367974 140191300101952 af.py:224] 2022-01-20 18:33:55.139522: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.368015 140191300101952 af.py:224] 2022-01-20 18:33:55.139528: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368056 140191300101952 af.py:224] 2022-01-20 18:33:55.139535: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368097 140191300101952 af.py:224] 2022-01-20 18:33:55.139539: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.368143 140191300101952 af.py:224] 2022-01-20 18:33:55.139546: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368233 140191300101952 af.py:224] 2022-01-20 18:33:55.139551: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.368274 140191300101952 af.py:224] 2022-01-20 18:33:55.139559: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.368315 140191300101952 af.py:224] 2022-01-20 18:33:55.139567: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368355 140191300101952 af.py:224] 2022-01-20 18:33:55.139573: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368396 140191300101952 af.py:224] 2022-01-20 18:33:55.139578: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.368437 140191300101952 af.py:224] 2022-01-20 18:33:55.139585: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368496 140191300101952 af.py:224] 2022-01-20 18:33:55.139590: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.368538 140191300101952 af.py:224] 2022-01-20 18:33:55.139599: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.368579 140191300101952 af.py:224] 2022-01-20 18:33:55.139606: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368620 140191300101952 af.py:224] 2022-01-20 18:33:55.139613: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368662 140191300101952 af.py:224] 2022-01-20 18:33:55.139618: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.368707 140191300101952 af.py:224] 2022-01-20 18:33:55.139624: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368749 140191300101952 af.py:224] 2022-01-20 18:33:55.139629: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.368789 140191300101952 af.py:224] 2022-01-20 18:33:55.139638: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.368829 140191300101952 af.py:224] 2022-01-20 18:33:55.139645: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368870 140191300101952 af.py:224] 2022-01-20 18:33:55.139651: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368911 140191300101952 af.py:224] 2022-01-20 18:33:55.139656: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.368952 140191300101952 af.py:224] 2022-01-20 18:33:55.139663: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.368992 140191300101952 af.py:224] 2022-01-20 18:33:55.139668: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.369032 140191300101952 af.py:224] 2022-01-20 18:33:55.139676: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.369073 140191300101952 af.py:224] 2022-01-20 18:33:55.139682: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369114 140191300101952 af.py:224] 2022-01-20 18:33:55.139689: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369155 140191300101952 af.py:224] 2022-01-20 18:33:55.139694: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.369196 140191300101952 af.py:224] 2022-01-20 18:33:55.139700: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369236 140191300101952 af.py:224] 2022-01-20 18:33:55.139705: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.369277 140191300101952 af.py:224] 2022-01-20 18:33:55.139714: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.369317 140191300101952 af.py:224] 2022-01-20 18:33:55.139721: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369357 140191300101952 af.py:224] 2022-01-20 18:33:55.139727: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369400 140191300101952 af.py:224] 2022-01-20 18:33:55.139732: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.369441 140191300101952 af.py:224] 2022-01-20 18:33:55.139739: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369482 140191300101952 af.py:224] 2022-01-20 18:33:55.139744: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.369523 140191300101952 af.py:224] 2022-01-20 18:33:55.139753: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.369563 140191300101952 af.py:224] 2022-01-20 18:33:55.139760: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369605 140191300101952 af.py:224] 2022-01-20 18:33:55.139767: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369645 140191300101952 af.py:224] 2022-01-20 18:33:55.139772: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.369686 140191300101952 af.py:224] 2022-01-20 18:33:55.139779: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369726 140191300101952 af.py:224] 2022-01-20 18:33:55.139784: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.369766 140191300101952 af.py:224] 2022-01-20 18:33:55.139794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.369807 140191300101952 af.py:224] 2022-01-20 18:33:55.139801: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369848 140191300101952 af.py:224] 2022-01-20 18:33:55.139808: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369888 140191300101952 af.py:224] 2022-01-20 18:33:55.139813: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.369928 140191300101952 af.py:224] 2022-01-20 18:33:55.139820: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.369969 140191300101952 af.py:224] 2022-01-20 18:33:55.139825: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.370010 140191300101952 af.py:224] 2022-01-20 18:33:55.139835: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.370051 140191300101952 af.py:224] 2022-01-20 18:33:55.139842: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370109 140191300101952 af.py:224] 2022-01-20 18:33:55.139848: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370151 140191300101952 af.py:224] 2022-01-20 18:33:55.139853: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.370193 140191300101952 af.py:224] 2022-01-20 18:33:55.139860: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370234 140191300101952 af.py:224] 2022-01-20 18:33:55.139865: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.370275 140191300101952 af.py:224] 2022-01-20 18:33:55.139874: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.370316 140191300101952 af.py:224] 2022-01-20 18:33:55.139881: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370357 140191300101952 af.py:224] 2022-01-20 18:33:55.139888: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370398 140191300101952 af.py:224] 2022-01-20 18:33:55.139893: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.370439 140191300101952 af.py:224] 2022-01-20 18:33:55.139900: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370479 140191300101952 af.py:224] 2022-01-20 18:33:55.139905: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.370520 140191300101952 af.py:224] 2022-01-20 18:33:55.139915: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.370560 140191300101952 af.py:224] 2022-01-20 18:33:55.139923: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370601 140191300101952 af.py:224] 2022-01-20 18:33:55.139929: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370642 140191300101952 af.py:224] 2022-01-20 18:33:55.139935: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.370683 140191300101952 af.py:224] 2022-01-20 18:33:55.139941: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370724 140191300101952 af.py:224] 2022-01-20 18:33:55.139947: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.370769 140191300101952 af.py:224] 2022-01-20 18:33:55.139955: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.370811 140191300101952 af.py:224] 2022-01-20 18:33:55.139962: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370852 140191300101952 af.py:224] 2022-01-20 18:33:55.139969: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370893 140191300101952 af.py:224] 2022-01-20 18:33:55.139974: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.370933 140191300101952 af.py:224] 2022-01-20 18:33:55.139981: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.370975 140191300101952 af.py:224] 2022-01-20 18:33:55.139986: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.371016 140191300101952 af.py:224] 2022-01-20 18:33:55.139995: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.371056 140191300101952 af.py:224] 2022-01-20 18:33:55.140002: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371097 140191300101952 af.py:224] 2022-01-20 18:33:55.140009: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371137 140191300101952 af.py:224] 2022-01-20 18:33:55.140013: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.371178 140191300101952 af.py:224] 2022-01-20 18:33:55.140020: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371219 140191300101952 af.py:224] 2022-01-20 18:33:55.140025: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.371259 140191300101952 af.py:224] 2022-01-20 18:33:55.140034: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.371300 140191300101952 af.py:224] 2022-01-20 18:33:55.140041: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371340 140191300101952 af.py:224] 2022-01-20 18:33:55.140047: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371381 140191300101952 af.py:224] 2022-01-20 18:33:55.140052: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.371422 140191300101952 af.py:224] 2022-01-20 18:33:55.140059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371465 140191300101952 af.py:224] 2022-01-20 18:33:55.140064: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.371506 140191300101952 af.py:224] 2022-01-20 18:33:55.140073: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.371547 140191300101952 af.py:224] 2022-01-20 18:33:55.140080: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371587 140191300101952 af.py:224] 2022-01-20 18:33:55.140087: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371628 140191300101952 af.py:224] 2022-01-20 18:33:55.140092: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.371669 140191300101952 af.py:224] 2022-01-20 18:33:55.140098: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371728 140191300101952 af.py:224] 2022-01-20 18:33:55.140103: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.371771 140191300101952 af.py:224] 2022-01-20 18:33:55.140112: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.371812 140191300101952 af.py:224] 2022-01-20 18:33:55.140123: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371853 140191300101952 af.py:224] 2022-01-20 18:33:55.140130: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371895 140191300101952 af.py:224] 2022-01-20 18:33:55.140135: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.371935 140191300101952 af.py:224] 2022-01-20 18:33:55.140142: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.371976 140191300101952 af.py:224] 2022-01-20 18:33:55.140147: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.372017 140191300101952 af.py:224] 2022-01-20 18:33:55.140156: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.372058 140191300101952 af.py:224] 2022-01-20 18:33:55.140164: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372098 140191300101952 af.py:224] 2022-01-20 18:33:55.140170: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372181 140191300101952 af.py:224] 2022-01-20 18:33:55.140175: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.372242 140191300101952 af.py:224] 2022-01-20 18:33:55.140182: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372284 140191300101952 af.py:224] 2022-01-20 18:33:55.140187: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.372325 140191300101952 af.py:224] 2022-01-20 18:33:55.140196: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.372366 140191300101952 af.py:224] 2022-01-20 18:33:55.140203: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372406 140191300101952 af.py:224] 2022-01-20 18:33:55.140209: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372447 140191300101952 af.py:224] 2022-01-20 18:33:55.140214: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.372488 140191300101952 af.py:224] 2022-01-20 18:33:55.140220: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372529 140191300101952 af.py:224] 2022-01-20 18:33:55.140225: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.372570 140191300101952 af.py:224] 2022-01-20 18:33:55.140234: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.372611 140191300101952 af.py:224] 2022-01-20 18:33:55.140241: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372652 140191300101952 af.py:224] 2022-01-20 18:33:55.140248: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372692 140191300101952 af.py:224] 2022-01-20 18:33:55.140253: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.372733 140191300101952 af.py:224] 2022-01-20 18:33:55.140260: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372774 140191300101952 af.py:224] 2022-01-20 18:33:55.140265: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.372815 140191300101952 af.py:224] 2022-01-20 18:33:55.140274: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.372855 140191300101952 af.py:224] 2022-01-20 18:33:55.140281: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372896 140191300101952 af.py:224] 2022-01-20 18:33:55.140287: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.372940 140191300101952 af.py:224] 2022-01-20 18:33:55.140292: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.372981 140191300101952 af.py:224] 2022-01-20 18:33:55.140300: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.373023 140191300101952 af.py:224] 2022-01-20 18:33:55.140305: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.373063 140191300101952 af.py:224] 2022-01-20 18:33:55.140314: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.373104 140191300101952 af.py:224] 2022-01-20 18:33:55.140321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.373145 140191300101952 af.py:224] 2022-01-20 18:33:55.140328: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.373186 140191300101952 af.py:224] 2022-01-20 18:33:55.140333: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.373227 140191300101952 af.py:224] 2022-01-20 18:33:55.140339: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.373268 140191300101952 af.py:224] 2022-01-20 18:33:55.140345: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.373309 140191300101952 af.py:224] 2022-01-20 18:33:55.140354: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.373350 140191300101952 af.py:224] 2022-01-20 18:33:55.140361: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.373409 140191300101952 af.py:224] 2022-01-20 18:33:55.140367: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.373451 140191300101952 af.py:224] 2022-01-20 18:33:55.140372: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.373492 140191300101952 af.py:224] 2022-01-20 18:33:55.140379: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5582a82c8bd0: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
I0120 10:33:55.373533 140191300101952 af.py:224] 2022-01-20 18:33:55.140384: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
I0120 10:33:55.373574 140191300101952 af.py:224] 2022-01-20 18:33:55.140393: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
I0120 10:33:55.373615 140191300101952 af.py:224] 2022-01-20 18:33:55.177101: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:1047] could not synchronize on CUDA context: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered :: *** Begin stack trace ***
I0120 10:33:55.373660 140191300101952 af.py:224] 
I0120 10:33:55.373701 140191300101952 af.py:224] 
I0120 10:33:55.373742 140191300101952 af.py:224] 
I0120 10:33:55.373783 140191300101952 af.py:224] 
I0120 10:33:55.373824 140191300101952 af.py:224] 
I0120 10:33:55.373864 140191300101952 af.py:224] 
I0120 10:33:55.373905 140191300101952 af.py:224] 
I0120 10:33:55.373945 140191300101952 af.py:224] 
I0120 10:33:55.373985 140191300101952 af.py:224] 
I0120 10:33:55.374025 140191300101952 af.py:224] 
I0120 10:33:55.374065 140191300101952 af.py:224] 
I0120 10:33:55.374105 140191300101952 af.py:224] 
I0120 10:33:55.374145 140191300101952 af.py:224] 
I0120 10:33:55.374185 140191300101952 af.py:224] 
I0120 10:33:55.374225 140191300101952 af.py:224] 
I0120 10:33:55.374264 140191300101952 af.py:224] 
I0120 10:33:55.374305 140191300101952 af.py:224] 
I0120 10:33:55.374345 140191300101952 af.py:224] 
I0120 10:33:55.374385 140191300101952 af.py:224] 
I0120 10:33:55.374425 140191300101952 af.py:224] 
I0120 10:33:55.374465 140191300101952 af.py:224] 
I0120 10:33:55.374505 140191300101952 af.py:224] 
I0120 10:33:55.374545 140191300101952 af.py:224] 
I0120 10:33:55.374585 140191300101952 af.py:224] _PyModule_ClearDict
I0120 10:33:55.374626 140191300101952 af.py:224] PyImport_Cleanup
I0120 10:33:55.374666 140191300101952 af.py:224] Py_FinalizeEx
I0120 10:33:55.374707 140191300101952 af.py:224] 
I0120 10:33:55.374747 140191300101952 af.py:224] _Py_UnixMain
I0120 10:33:55.374788 140191300101952 af.py:224] __libc_start_main
I0120 10:33:55.374828 140191300101952 af.py:224] 
I0120 10:33:55.374869 140191300101952 af.py:224] *** End stack trace ***
I0120 10:33:55.374910 140191300101952 af.py:224] 
I0120 10:33:55.374950 140191300101952 af.py:224] 2022-01-20 18:33:55.177141: F external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_executable.cc:141] Check failed: pair.first->SynchronizeAllActivity()
I0120 10:33:55.374991 140191300101952 af.py:224] Fatal Python error: Aborted
I0120 10:33:55.375033 140191300101952 af.py:224] 
I0120 10:33:55.375074 140191300101952 af.py:224] Current thread 0x00007f4d184670c0 (most recent call first):
I0120 10:33:57.551114 140191300101952 af.py:224] /app/run_alphafold.sh: line 3:     8 Aborted                 (core dumped) python /app/alphafold/run_alphafold.py "$@"