{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "6XvCUmCEd4Dm" }, "source": [ "# TensorFlow Datasets\n", "\n", "TFDS provides a collection of ready-to-use datasets for use with TensorFlow, Jax, and other Machine Learning frameworks.\n", "\n", "It handles downloading and preparing the data deterministically and constructing a `tf.data.Dataset` (or `np.array`).\n", "\n", "Note: Do not confuse [TFDS](https://www.tensorflow.org/datasets) (this library) with `tf.data` (TensorFlow API to build efficient data pipelines). TFDS is a high level wrapper around `tf.data`. If you're not familiar with this API, we encourage you to read [the official tf.data guide](https://www.tensorflow.org/guide/data) first.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "J8y9ZkLXmAZc" }, "source": [ "Copyright 2018 The TensorFlow Datasets Authors, Licensed under the Apache License, Version 2.0" ] }, { "cell_type": "markdown", "metadata": { "id": "OGw9EgE0tC0C" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "_7hshda5eaGL" }, "source": [ "## Installation\n", "\n", "TFDS exists in two packages:\n", "\n", "* `pip install tensorflow-datasets`: The stable version, released every few months.\n", "* `pip install tfds-nightly`: Released every day, contains the last versions of the datasets.\n", "\n", "This colab uses `tfds-nightly`:\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "both", "execution": { "iopub.execute_input": "2024-12-14T12:41:48.815135Z", "iopub.status.busy": "2024-12-14T12:41:48.814610Z", "iopub.status.idle": "2024-12-14T12:41:52.490191Z", "shell.execute_reply": "2024-12-14T12:41:52.489105Z" }, "id": "boeZp0sYbO41" }, "outputs": [], "source": [ "!pip install -q tfds-nightly tensorflow matplotlib" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:41:52.494224Z", "iopub.status.busy": "2024-12-14T12:41:52.493604Z", "iopub.status.idle": "2024-12-14T12:41:55.771125Z", "shell.execute_reply": "2024-12-14T12:41:55.770280Z" }, "id": "TTBSvHcSLBzc" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-12-14 12:41:53.251913: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", "E0000 00:00:1734180113.275388 782567 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", "E0000 00:00:1734180113.282524 782567 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n" ] } ], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import tensorflow as tf\n", "\n", "import tensorflow_datasets as tfds" ] }, { "cell_type": "markdown", "metadata": { "id": "VZZyuO13fPvk" }, "source": [ "## Find available datasets\n", "\n", "All dataset builders are subclass of `tfds.core.DatasetBuilder`. To get the list of available builders, use `tfds.list_builders()` or look at our [catalog](https://www.tensorflow.org/datasets/catalog/overview)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:41:55.775102Z", "iopub.status.busy": "2024-12-14T12:41:55.774216Z", "iopub.status.idle": "2024-12-14T12:41:58.188788Z", "shell.execute_reply": "2024-12-14T12:41:58.188021Z" }, "id": "FAvbSVzjLCIb" }, "outputs": [ { "data": { "text/plain": [ "['abstract_reasoning',\n", " 'accentdb',\n", " 'aeslc',\n", " 'aflw2k3d',\n", " 'ag_news_subset',\n", " 'ai2_arc',\n", " 'ai2_arc_with_ir',\n", " 'amazon_us_reviews',\n", " 'anli',\n", " 'answer_equivalence',\n", " 'arc',\n", " 'asqa',\n", " 'asset',\n", " 'assin2',\n", " 'asu_table_top_converted_externally_to_rlds',\n", " 'austin_buds_dataset_converted_externally_to_rlds',\n", " 'austin_sailor_dataset_converted_externally_to_rlds',\n", " 'austin_sirius_dataset_converted_externally_to_rlds',\n", " 'bair_robot_pushing_small',\n", " 'bc_z',\n", " 'bccd',\n", " 'beans',\n", " 'bee_dataset',\n", " 'beir',\n", " 'berkeley_autolab_ur5',\n", " 'berkeley_cable_routing',\n", " 'berkeley_fanuc_manipulation',\n", " 'berkeley_gnm_cory_hall',\n", " 'berkeley_gnm_recon',\n", " 'berkeley_gnm_sac_son',\n", " 'berkeley_mvp_converted_externally_to_rlds',\n", " 'berkeley_rpt_converted_externally_to_rlds',\n", " 'big_patent',\n", " 'bigearthnet',\n", " 'billsum',\n", " 'binarized_mnist',\n", " 'binary_alpha_digits',\n", " 'ble_wind_field',\n", " 'blimp',\n", " 'booksum',\n", " 'bool_q',\n", " 'bot_adversarial_dialogue',\n", " 'bridge',\n", " 'bucc',\n", " 'c4',\n", " 'c4_wsrs',\n", " 'caltech101',\n", " 'caltech_birds2010',\n", " 'caltech_birds2011',\n", " 'cardiotox',\n", " 'cars196',\n", " 'cassava',\n", " 'cats_vs_dogs',\n", " 'celeb_a',\n", " 'celeb_a_hq',\n", " 'cfq',\n", " 'cherry_blossoms',\n", " 'chexpert',\n", " 'cifar10',\n", " 'cifar100',\n", " 'cifar100_n',\n", " 'cifar10_1',\n", " 'cifar10_corrupted',\n", " 'cifar10_h',\n", " 'cifar10_n',\n", " 'citrus_leaves',\n", " 'cityscapes',\n", " 'civil_comments',\n", " 'clevr',\n", " 'clic',\n", " 'clinc_oos',\n", " 'cmaterdb',\n", " 'cmu_franka_exploration_dataset_converted_externally_to_rlds',\n", " 'cmu_play_fusion',\n", " 'cmu_stretch',\n", " 'cnn_dailymail',\n", " 'coco',\n", " 'coco_captions',\n", " 'coil100',\n", " 'colorectal_histology',\n", " 'colorectal_histology_large',\n", " 'columbia_cairlab_pusht_real',\n", " 'common_voice',\n", " 'conll2002',\n", " 'conll2003',\n", " 'controlled_noisy_web_labels',\n", " 'coqa',\n", " 'corr2cause',\n", " 'cos_e',\n", " 'cosmos_qa',\n", " 'covid19',\n", " 'covid19sum',\n", " 'crema_d',\n", " 'criteo',\n", " 'cs_restaurants',\n", " 'curated_breast_imaging_ddsm',\n", " 'cycle_gan',\n", " 'd4rl_adroit_door',\n", " 'd4rl_adroit_hammer',\n", " 'd4rl_adroit_pen',\n", " 'd4rl_adroit_relocate',\n", " 'd4rl_antmaze',\n", " 'd4rl_mujoco_ant',\n", " 'd4rl_mujoco_halfcheetah',\n", " 'd4rl_mujoco_hopper',\n", " 'd4rl_mujoco_walker2d',\n", " 'dart',\n", " 'databricks_dolly',\n", " 'davis',\n", " 'deep1b',\n", " 'deep_weeds',\n", " 'definite_pronoun_resolution',\n", " 'dementiabank',\n", " 'diabetic_retinopathy_detection',\n", " 'diamonds',\n", " 'div2k',\n", " 'dlr_edan_shared_control_converted_externally_to_rlds',\n", " 'dlr_sara_grid_clamp_converted_externally_to_rlds',\n", " 'dlr_sara_pour_converted_externally_to_rlds',\n", " 'dmlab',\n", " 'doc_nli',\n", " 'dolphin_number_word',\n", " 'domainnet',\n", " 'downsampled_imagenet',\n", " 'drop',\n", " 'dsprites',\n", " 'dtd',\n", " 'duke_ultrasound',\n", " 'e2e_cleaned',\n", " 'efron_morris75',\n", " 'emnist',\n", " 'eraser_multi_rc',\n", " 'esnli',\n", " 'eth_agent_affordances',\n", " 'eurosat',\n", " 'fashion_mnist',\n", " 'flic',\n", " 'flores',\n", " 'food101',\n", " 'forest_fires',\n", " 'fractal20220817_data',\n", " 'fuss',\n", " 'gap',\n", " 'geirhos_conflict_stimuli',\n", " 'gem',\n", " 'genomics_ood',\n", " 'german_credit_numeric',\n", " 'gigaword',\n", " 'glove100_angular',\n", " 'glue',\n", " 'goemotions',\n", " 'gov_report',\n", " 'gpt3',\n", " 'gref',\n", " 'groove',\n", " 'grounded_scan',\n", " 'gsm8k',\n", " 'gtzan',\n", " 'gtzan_music_speech',\n", " 'hellaswag',\n", " 'higgs',\n", " 'hillstrom',\n", " 'horses_or_humans',\n", " 'howell',\n", " 'i_naturalist2017',\n", " 'i_naturalist2018',\n", " 'i_naturalist2021',\n", " 'iamlab_cmu_pickup_insert_converted_externally_to_rlds',\n", " 'imagenet2012',\n", " 'imagenet2012_corrupted',\n", " 'imagenet2012_fewshot',\n", " 'imagenet2012_multilabel',\n", " 'imagenet2012_real',\n", " 'imagenet2012_subset',\n", " 'imagenet_a',\n", " 'imagenet_lt',\n", " 'imagenet_pi',\n", " 'imagenet_r',\n", " 'imagenet_resized',\n", " 'imagenet_sketch',\n", " 'imagenet_v2',\n", " 'imagenette',\n", " 'imagewang',\n", " 'imdb_reviews',\n", " 'imperialcollege_sawyer_wrist_cam',\n", " 'irc_disentanglement',\n", " 'iris',\n", " 'istella',\n", " 'jaco_play',\n", " 'kaist_nonprehensile_converted_externally_to_rlds',\n", " 'kddcup99',\n", " 'kitti',\n", " 'kmnist',\n", " 'kuka',\n", " 'laion400m',\n", " 'lambada',\n", " 'lfw',\n", " 'librispeech',\n", " 'librispeech_lm',\n", " 'libritts',\n", " 'ljspeech',\n", " 'lm1b',\n", " 'locomotion',\n", " 'lost_and_found',\n", " 'lsun',\n", " 'lvis',\n", " 'malaria',\n", " 'maniskill_dataset_converted_externally_to_rlds',\n", " 'math_dataset',\n", " 'math_qa',\n", " 'mctaco',\n", " 'media_sum',\n", " 'mlqa',\n", " 'mnist',\n", " 'mnist_corrupted',\n", " 'movie_lens',\n", " 'movie_rationales',\n", " 'movielens',\n", " 'moving_mnist',\n", " 'mrqa',\n", " 'mslr_web',\n", " 'mt_opt',\n", " 'mtnt',\n", " 'multi_news',\n", " 'multi_nli',\n", " 'multi_nli_mismatch',\n", " 'natural_instructions',\n", " 'natural_questions',\n", " 'natural_questions_open',\n", " 'newsroom',\n", " 'nsynth',\n", " 'nyu_depth_v2',\n", " 'nyu_door_opening_surprising_effectiveness',\n", " 'nyu_franka_play_dataset_converted_externally_to_rlds',\n", " 'nyu_rot_dataset_converted_externally_to_rlds',\n", " 'ogbg_molpcba',\n", " 'omniglot',\n", " 'open_images_challenge2019_detection',\n", " 'open_images_v4',\n", " 'openbookqa',\n", " 'opinion_abstracts',\n", " 'opinosis',\n", " 'opus',\n", " 'oxford_flowers102',\n", " 'oxford_iiit_pet',\n", " 'para_crawl',\n", " 'pass',\n", " 'patch_camelyon',\n", " 'paws_wiki',\n", " 'paws_x_wiki',\n", " 'penguins',\n", " 'pet_finder',\n", " 'pg19',\n", " 'piqa',\n", " 'places365_small',\n", " 'placesfull',\n", " 'plant_leaves',\n", " 'plant_village',\n", " 'plantae_k',\n", " 'protein_net',\n", " 'q_re_cc',\n", " 'qa4mre',\n", " 'qasc',\n", " 'quac',\n", " 'quality',\n", " 'quickdraw_bitmap',\n", " 'race',\n", " 'radon',\n", " 'real_toxicity_prompts',\n", " 'reddit',\n", " 'reddit_disentanglement',\n", " 'reddit_tifu',\n", " 'ref_coco',\n", " 'resisc45',\n", " 'rlu_atari',\n", " 'rlu_atari_checkpoints',\n", " 'rlu_atari_checkpoints_ordered',\n", " 'rlu_control_suite',\n", " 'rlu_dmlab_explore_object_rewards_few',\n", " 'rlu_dmlab_explore_object_rewards_many',\n", " 'rlu_dmlab_rooms_select_nonmatching_object',\n", " 'rlu_dmlab_rooms_watermaze',\n", " 'rlu_dmlab_seekavoid_arena01',\n", " 'rlu_locomotion',\n", " 'rlu_rwrl',\n", " 'robomimic_mg',\n", " 'robomimic_mh',\n", " 'robomimic_ph',\n", " 'robonet',\n", " 'robosuite_panda_pick_place_can',\n", " 'roboturk',\n", " 'rock_paper_scissors',\n", " 'rock_you',\n", " 's3o4d',\n", " 'salient_span_wikipedia',\n", " 'samsum',\n", " 'savee',\n", " 'scan',\n", " 'scene_parse150',\n", " 'schema_guided_dialogue',\n", " 'sci_tail',\n", " 'scicite',\n", " 'scientific_papers',\n", " 'scrolls',\n", " 'segment_anything',\n", " 'sentiment140',\n", " 'shapes3d',\n", " 'sift1m',\n", " 'simpte',\n", " 'siscore',\n", " 'smallnorb',\n", " 'smartwatch_gestures',\n", " 'snli',\n", " 'so2sat',\n", " 'speech_commands',\n", " 'spoken_digit',\n", " 'squad',\n", " 'squad_question_generation',\n", " 'stanford_dogs',\n", " 'stanford_hydra_dataset_converted_externally_to_rlds',\n", " 'stanford_kuka_multimodal_dataset_converted_externally_to_rlds',\n", " 'stanford_mask_vit_converted_externally_to_rlds',\n", " 'stanford_online_products',\n", " 'stanford_robocook_converted_externally_to_rlds',\n", " 'star_cfq',\n", " 'starcraft_video',\n", " 'stl10',\n", " 'story_cloze',\n", " 'summscreen',\n", " 'sun397',\n", " 'super_glue',\n", " 'svhn_cropped',\n", " 'symmetric_solids',\n", " 'taco_play',\n", " 'tao',\n", " 'tatoeba',\n", " 'ted_hrlr_translate',\n", " 'ted_multi_translate',\n", " 'tedlium',\n", " 'tf_flowers',\n", " 'the300w_lp',\n", " 'tiny_shakespeare',\n", " 'titanic',\n", " 'tokyo_u_lsmo_converted_externally_to_rlds',\n", " 'toto',\n", " 'trec',\n", " 'trivia_qa',\n", " 'tydi_qa',\n", " 'uc_merced',\n", " 'ucf101',\n", " 'ucsd_kitchen_dataset_converted_externally_to_rlds',\n", " 'ucsd_pick_and_place_dataset_converted_externally_to_rlds',\n", " 'uiuc_d3field',\n", " 'unified_qa',\n", " 'universal_dependencies',\n", " 'unnatural_instructions',\n", " 'usc_cloth_sim_converted_externally_to_rlds',\n", " 'user_libri_audio',\n", " 'user_libri_text',\n", " 'utaustin_mutex',\n", " 'utokyo_pr2_opening_fridge_converted_externally_to_rlds',\n", " 'utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds',\n", " 'utokyo_saytap_converted_externally_to_rlds',\n", " 'utokyo_xarm_bimanual_converted_externally_to_rlds',\n", " 'utokyo_xarm_pick_and_place_converted_externally_to_rlds',\n", " 'vctk',\n", " 'viola',\n", " 'visual_domain_decathlon',\n", " 'voc',\n", " 'voxceleb',\n", " 'voxforge',\n", " 'waymo_open_dataset',\n", " 'web_graph',\n", " 'web_nlg',\n", " 'web_questions',\n", " 'webvid',\n", " 'wider_face',\n", " 'wiki40b',\n", " 'wiki_auto',\n", " 'wiki_bio',\n", " 'wiki_dialog',\n", " 'wiki_table_questions',\n", " 'wiki_table_text',\n", " 'wikiann',\n", " 'wikihow',\n", " 'wikipedia',\n", " 'wikipedia_toxicity_subtypes',\n", " 'wine_quality',\n", " 'winogrande',\n", " 'wit',\n", " 'wit_kaggle',\n", " 'wmt13_translate',\n", " 'wmt14_translate',\n", " 'wmt15_translate',\n", " 'wmt16_translate',\n", " 'wmt17_translate',\n", " 'wmt18_translate',\n", " 'wmt19_translate',\n", " 'wmt_t2t_translate',\n", " 'wmt_translate',\n", " 'wordnet',\n", " 'wsc273',\n", " 'xnli',\n", " 'xquad',\n", " 'xsum',\n", " 'xtreme_pawsx',\n", " 'xtreme_pos',\n", " 'xtreme_s',\n", " 'xtreme_xnli',\n", " 'yahoo_ltrc',\n", " 'yelp_polarity_reviews',\n", " 'yes_no',\n", " 'youtube_vis',\n", " 'huggingface:acronym_identification',\n", " 'huggingface:ade_corpus_v2',\n", " 'huggingface:adv_glue',\n", " 'huggingface:adversarial_qa',\n", " 'huggingface:aeslc',\n", " 'huggingface:afrikaans_ner_corpus',\n", " 'huggingface:ag_news',\n", " 'huggingface:ai2_arc',\n", " 'huggingface:air_dialogue',\n", " 'huggingface:ajgt_twitter_ar',\n", " 'huggingface:allegro_reviews',\n", " 'huggingface:allocine',\n", " 'huggingface:alt',\n", " 'huggingface:amazon_polarity',\n", " 'huggingface:amazon_reviews_multi',\n", " 'huggingface:amazon_us_reviews',\n", " 'huggingface:ambig_qa',\n", " 'huggingface:americas_nli',\n", " 'huggingface:ami',\n", " 'huggingface:amttl',\n", " 'huggingface:anli',\n", " 'huggingface:app_reviews',\n", " 'huggingface:aqua_rat',\n", " 'huggingface:aquamuse',\n", " 'huggingface:ar_cov19',\n", " 'huggingface:ar_res_reviews',\n", " 'huggingface:ar_sarcasm',\n", " 'huggingface:arabic_billion_words',\n", " 'huggingface:arabic_pos_dialect',\n", " 'huggingface:arabic_speech_corpus',\n", " 'huggingface:arcd',\n", " 'huggingface:arsentd_lev',\n", " 'huggingface:art',\n", " 'huggingface:arxiv_dataset',\n", " 'huggingface:ascent_kb',\n", " 'huggingface:aslg_pc12',\n", " 'huggingface:asnq',\n", " 'huggingface:asset',\n", " 'huggingface:assin',\n", " 'huggingface:assin2',\n", " 'huggingface:atomic',\n", " 'huggingface:autshumato',\n", " 'huggingface:babi_qa',\n", " 'huggingface:banking77',\n", " 'huggingface:bbaw_egyptian',\n", " 'huggingface:bbc_hindi_nli',\n", " 'huggingface:bc2gm_corpus',\n", " 'huggingface:beans',\n", " 'huggingface:best2009',\n", " 'huggingface:bianet',\n", " 'huggingface:bible_para',\n", " 'huggingface:big_patent',\n", " 'huggingface:bigbench',\n", " 'huggingface:billsum',\n", " 'huggingface:bing_coronavirus_query_set',\n", " 'huggingface:biomrc',\n", " 'huggingface:biosses',\n", " 'huggingface:biwi_kinect_head_pose',\n", " 'huggingface:blbooks',\n", " 'huggingface:blbooksgenre',\n", " 'huggingface:blended_skill_talk',\n", " 'huggingface:blimp',\n", " 'huggingface:blog_authorship_corpus',\n", " 'huggingface:bn_hate_speech',\n", " 'huggingface:bnl_newspapers',\n", " 'huggingface:bookcorpus',\n", " 'huggingface:bookcorpusopen',\n", " 'huggingface:boolq',\n", " 'huggingface:bprec',\n", " 'huggingface:break_data',\n", " 'huggingface:brwac',\n", " 'huggingface:bsd_ja_en',\n", " 'huggingface:bswac',\n", " 'huggingface:c3',\n", " 'huggingface:c4',\n", " 'huggingface:cail2018',\n", " 'huggingface:caner',\n", " 'huggingface:capes',\n", " 'huggingface:casino',\n", " 'huggingface:catalonia_independence',\n", " 'huggingface:cats_vs_dogs',\n", " 'huggingface:cawac',\n", " 'huggingface:cbt',\n", " 'huggingface:cc100',\n", " 'huggingface:cc_news',\n", " 'huggingface:ccaligned_multilingual',\n", " 'huggingface:cdsc',\n", " 'huggingface:cdt',\n", " 'huggingface:cedr',\n", " 'huggingface:cfq',\n", " 'huggingface:chr_en',\n", " 'huggingface:cifar10',\n", " 'huggingface:cifar100',\n", " 'huggingface:circa',\n", " 'huggingface:civil_comments',\n", " 'huggingface:clickbait_news_bg',\n", " 'huggingface:climate_fever',\n", " 'huggingface:clinc_oos',\n", " 'huggingface:clue',\n", " 'huggingface:cmrc2018',\n", " 'huggingface:cmu_hinglish_dog',\n", " 'huggingface:cnn_dailymail',\n", " 'huggingface:coached_conv_pref',\n", " 'huggingface:coarse_discourse',\n", " 'huggingface:codah',\n", " 'huggingface:code_search_net',\n", " 'huggingface:code_x_glue_cc_clone_detection_big_clone_bench',\n", " 'huggingface:code_x_glue_cc_clone_detection_poj104',\n", " 'huggingface:code_x_glue_cc_cloze_testing_all',\n", " 'huggingface:code_x_glue_cc_cloze_testing_maxmin',\n", " 'huggingface:code_x_glue_cc_code_completion_line',\n", " 'huggingface:code_x_glue_cc_code_completion_token',\n", " 'huggingface:code_x_glue_cc_code_refinement',\n", " 'huggingface:code_x_glue_cc_code_to_code_trans',\n", " 'huggingface:code_x_glue_cc_defect_detection',\n", " 'huggingface:code_x_glue_ct_code_to_text',\n", " 'huggingface:code_x_glue_tc_nl_code_search_adv',\n", " 'huggingface:code_x_glue_tc_text_to_code',\n", " 'huggingface:code_x_glue_tt_text_to_text',\n", " 'huggingface:com_qa',\n", " 'huggingface:common_gen',\n", " 'huggingface:common_language',\n", " 'huggingface:common_voice',\n", " 'huggingface:commonsense_qa',\n", " 'huggingface:competition_math',\n", " 'huggingface:compguesswhat',\n", " 'huggingface:conceptnet5',\n", " 'huggingface:conceptual_12m',\n", " 'huggingface:conceptual_captions',\n", " 'huggingface:conll2000',\n", " 'huggingface:conll2002',\n", " 'huggingface:conll2003',\n", " 'huggingface:conll2012_ontonotesv5',\n", " 'huggingface:conllpp',\n", " 'huggingface:consumer-finance-complaints',\n", " 'huggingface:conv_ai',\n", " 'huggingface:conv_ai_2',\n", " 'huggingface:conv_ai_3',\n", " 'huggingface:conv_questions',\n", " 'huggingface:coqa',\n", " 'huggingface:cord19',\n", " 'huggingface:cornell_movie_dialog',\n", " 'huggingface:cos_e',\n", " 'huggingface:cosmos_qa',\n", " 'huggingface:counter',\n", " 'huggingface:covid_qa_castorini',\n", " 'huggingface:covid_qa_deepset',\n", " 'huggingface:covid_qa_ucsd',\n", " 'huggingface:covid_tweets_japanese',\n", " 'huggingface:covost2',\n", " 'huggingface:cppe-5',\n", " 'huggingface:craigslist_bargains',\n", " 'huggingface:crawl_domain',\n", " 'huggingface:crd3',\n", " 'huggingface:crime_and_punish',\n", " 'huggingface:crows_pairs',\n", " 'huggingface:cryptonite',\n", " 'huggingface:cs_restaurants',\n", " 'huggingface:cuad',\n", " 'huggingface:curiosity_dialogs',\n", " 'huggingface:daily_dialog',\n", " 'huggingface:dane',\n", " 'huggingface:danish_political_comments',\n", " 'huggingface:dart',\n", " 'huggingface:datacommons_factcheck',\n", " 'huggingface:dbpedia_14',\n", " 'huggingface:dbrd',\n", " 'huggingface:deal_or_no_dialog',\n", " 'huggingface:definite_pronoun_resolution',\n", " 'huggingface:dengue_filipino',\n", " 'huggingface:dialog_re',\n", " 'huggingface:diplomacy_detection',\n", " 'huggingface:disaster_response_messages',\n", " 'huggingface:discofuse',\n", " 'huggingface:discovery',\n", " 'huggingface:disfl_qa',\n", " 'huggingface:doc2dial',\n", " 'huggingface:docred',\n", " 'huggingface:doqa',\n", " 'huggingface:dream',\n", " 'huggingface:drop',\n", " 'huggingface:duorc',\n", " 'huggingface:dutch_social',\n", " 'huggingface:dyk',\n", " 'huggingface:e2e_nlg',\n", " 'huggingface:e2e_nlg_cleaned',\n", " 'huggingface:ecb',\n", " 'huggingface:ecthr_cases',\n", " 'huggingface:eduge',\n", " 'huggingface:ehealth_kd',\n", " 'huggingface:eitb_parcc',\n", " 'huggingface:electricity_load_diagrams',\n", " 'huggingface:eli5',\n", " 'huggingface:eli5_category',\n", " 'huggingface:elkarhizketak',\n", " 'huggingface:emea',\n", " 'huggingface:emo',\n", " 'huggingface:emotion',\n", " 'huggingface:emotone_ar',\n", " 'huggingface:empathetic_dialogues',\n", " 'huggingface:enriched_web_nlg',\n", " 'huggingface:enwik8',\n", " 'huggingface:eraser_multi_rc',\n", " 'huggingface:esnli',\n", " 'huggingface:eth_py150_open',\n", " 'huggingface:ethos',\n", " 'huggingface:ett',\n", " 'huggingface:eu_regulatory_ir',\n", " 'huggingface:eurlex',\n", " 'huggingface:euronews',\n", " 'huggingface:europa_eac_tm',\n", " 'huggingface:europa_ecdc_tm',\n", " 'huggingface:europarl_bilingual',\n", " 'huggingface:event2Mind',\n", " 'huggingface:evidence_infer_treatment',\n", " 'huggingface:exams',\n", " 'huggingface:factckbr',\n", " 'huggingface:fake_news_english',\n", " 'huggingface:fake_news_filipino',\n", " 'huggingface:farsi_news',\n", " 'huggingface:fashion_mnist',\n", " 'huggingface:fever',\n", " 'huggingface:few_rel',\n", " 'huggingface:financial_phrasebank',\n", " 'huggingface:finer',\n", " 'huggingface:flores',\n", " 'huggingface:flue',\n", " 'huggingface:food101',\n", " 'huggingface:fquad',\n", " 'huggingface:freebase_qa',\n", " 'huggingface:gap',\n", " 'huggingface:gem',\n", " 'huggingface:generated_reviews_enth',\n", " 'huggingface:generics_kb',\n", " 'huggingface:german_legal_entity_recognition',\n", " 'huggingface:germaner',\n", " 'huggingface:germeval_14',\n", " 'huggingface:giga_fren',\n", " 'huggingface:gigaword',\n", " 'huggingface:glucose',\n", " 'huggingface:glue',\n", " 'huggingface:gnad10',\n", " 'huggingface:go_emotions',\n", " 'huggingface:gooaq',\n", " 'huggingface:google_wellformed_query',\n", " 'huggingface:grail_qa',\n", " 'huggingface:great_code',\n", " 'huggingface:greek_legal_code',\n", " 'huggingface:gsm8k',\n", " 'huggingface:guardian_authorship',\n", " 'huggingface:gutenberg_time',\n", " 'huggingface:hans',\n", " 'huggingface:hansards',\n", " 'huggingface:hard',\n", " 'huggingface:harem',\n", " 'huggingface:has_part',\n", " 'huggingface:hate_offensive',\n", " 'huggingface:hate_speech18',\n", " 'huggingface:hate_speech_filipino',\n", " 'huggingface:hate_speech_offensive',\n", " 'huggingface:hate_speech_pl',\n", " 'huggingface:hate_speech_portuguese',\n", " 'huggingface:hatexplain',\n", " 'huggingface:hausa_voa_ner',\n", " 'huggingface:hausa_voa_topics',\n", " 'huggingface:hda_nli_hindi',\n", " 'huggingface:head_qa',\n", " 'huggingface:health_fact',\n", " 'huggingface:hebrew_projectbenyehuda',\n", " 'huggingface:hebrew_sentiment',\n", " 'huggingface:hebrew_this_world',\n", " 'huggingface:hellaswag',\n", " 'huggingface:hendrycks_test',\n", " 'huggingface:hind_encorp',\n", " 'huggingface:hindi_discourse',\n", " 'huggingface:hippocorpus',\n", " 'huggingface:hkcancor',\n", " 'huggingface:hlgd',\n", " 'huggingface:hope_edi',\n", " 'huggingface:hotpot_qa',\n", " 'huggingface:hover',\n", " 'huggingface:hrenwac_para',\n", " 'huggingface:hrwac',\n", " 'huggingface:humicroedit',\n", " 'huggingface:hybrid_qa',\n", " 'huggingface:hyperpartisan_news_detection',\n", " 'huggingface:iapp_wiki_qa_squad',\n", " 'huggingface:id_clickbait',\n", " 'huggingface:id_liputan6',\n", " 'huggingface:id_nergrit_corpus',\n", " 'huggingface:id_newspapers_2018',\n", " 'huggingface:id_panl_bppt',\n", " 'huggingface:id_puisi',\n", " 'huggingface:igbo_english_machine_translation',\n", " 'huggingface:igbo_monolingual',\n", " 'huggingface:igbo_ner',\n", " 'huggingface:ilist',\n", " 'huggingface:imagenet-1k',\n", " 'huggingface:imagenet_sketch',\n", " 'huggingface:imdb',\n", " 'huggingface:imdb_urdu_reviews',\n", " 'huggingface:imppres',\n", " 'huggingface:indic_glue',\n", " 'huggingface:indonli',\n", " 'huggingface:indonlu',\n", " 'huggingface:inquisitive_qg',\n", " 'huggingface:interpress_news_category_tr',\n", " 'huggingface:interpress_news_category_tr_lite',\n", " 'huggingface:irc_disentangle',\n", " 'huggingface:isixhosa_ner_corpus',\n", " 'huggingface:isizulu_ner_corpus',\n", " 'huggingface:iwslt2017',\n", " 'huggingface:jeopardy',\n", " 'huggingface:jfleg',\n", " 'huggingface:jigsaw_toxicity_pred',\n", " 'huggingface:jigsaw_unintended_bias',\n", " 'huggingface:jnlpba',\n", " 'huggingface:journalists_questions',\n", " 'huggingface:kan_hope',\n", " 'huggingface:kannada_news',\n", " 'huggingface:kd_conv',\n", " 'huggingface:kde4',\n", " 'huggingface:kelm',\n", " 'huggingface:kilt_tasks',\n", " 'huggingface:kilt_wikipedia',\n", " 'huggingface:kinnews_kirnews',\n", " 'huggingface:klue',\n", " 'huggingface:kor_3i4k',\n", " 'huggingface:kor_hate',\n", " 'huggingface:kor_ner',\n", " 'huggingface:kor_nli',\n", " 'huggingface:kor_nlu',\n", " 'huggingface:kor_qpair',\n", " 'huggingface:kor_sae',\n", " 'huggingface:kor_sarcasm',\n", " 'huggingface:labr',\n", " 'huggingface:lama',\n", " 'huggingface:lambada',\n", " 'huggingface:large_spanish_corpus',\n", " 'huggingface:laroseda',\n", " 'huggingface:lc_quad',\n", " 'huggingface:lccc',\n", " 'huggingface:lener_br',\n", " 'huggingface:lex_glue',\n", " 'huggingface:liar',\n", " 'huggingface:librispeech_asr',\n", " 'huggingface:librispeech_lm',\n", " 'huggingface:limit',\n", " 'huggingface:lince',\n", " 'huggingface:linnaeus',\n", " 'huggingface:liveqa',\n", " 'huggingface:lj_speech',\n", " 'huggingface:lm1b',\n", " 'huggingface:lst20',\n", " 'huggingface:m_lama',\n", " 'huggingface:mac_morpho',\n", " 'huggingface:makhzan',\n", " 'huggingface:masakhaner',\n", " 'huggingface:math_dataset',\n", " 'huggingface:math_qa',\n", " 'huggingface:matinf',\n", " 'huggingface:mbpp',\n", " 'huggingface:mc4',\n", " 'huggingface:mc_taco',\n", " 'huggingface:md_gender_bias',\n", " 'huggingface:mdd',\n", " 'huggingface:med_hop',\n", " 'huggingface:medal',\n", " 'huggingface:medical_dialog',\n", " 'huggingface:medical_questions_pairs',\n", " 'huggingface:medmcqa',\n", " 'huggingface:menyo20k_mt',\n", " 'huggingface:meta_woz',\n", " 'huggingface:metashift',\n", " 'huggingface:metooma',\n", " 'huggingface:metrec',\n", " 'huggingface:miam',\n", " 'huggingface:mkb',\n", " 'huggingface:mkqa',\n", " 'huggingface:mlqa',\n", " 'huggingface:mlsum',\n", " 'huggingface:mnist',\n", " 'huggingface:mocha',\n", " 'huggingface:monash_tsf',\n", " 'huggingface:moroco',\n", " 'huggingface:movie_rationales',\n", " 'huggingface:mrqa',\n", " 'huggingface:ms_marco',\n", " 'huggingface:ms_terms',\n", " 'huggingface:msr_genomics_kbcomp',\n", " 'huggingface:msr_sqa',\n", " 'huggingface:msr_text_compression',\n", " 'huggingface:msr_zhen_translation_parity',\n", " 'huggingface:msra_ner',\n", " 'huggingface:mt_eng_vietnamese',\n", " 'huggingface:muchocine',\n", " 'huggingface:multi_booked',\n", " 'huggingface:multi_eurlex',\n", " 'huggingface:multi_news',\n", " 'huggingface:multi_nli',\n", " 'huggingface:multi_nli_mismatch',\n", " 'huggingface:multi_para_crawl',\n", " 'huggingface:multi_re_qa',\n", " 'huggingface:multi_woz_v22',\n", " 'huggingface:multi_x_science_sum',\n", " 'huggingface:multidoc2dial',\n", " 'huggingface:multilingual_librispeech',\n", " 'huggingface:mutual_friends',\n", " 'huggingface:mwsc',\n", " 'huggingface:myanmar_news',\n", " 'huggingface:narrativeqa',\n", " 'huggingface:narrativeqa_manual',\n", " 'huggingface:natural_questions',\n", " 'huggingface:ncbi_disease',\n", " 'huggingface:nchlt',\n", " 'huggingface:ncslgr',\n", " 'huggingface:nell',\n", " 'huggingface:neural_code_search',\n", " 'huggingface:news_commentary',\n", " 'huggingface:newsgroup',\n", " 'huggingface:newsph',\n", " 'huggingface:newsph_nli',\n", " 'huggingface:newspop',\n", " 'huggingface:newsqa',\n", " 'huggingface:newsroom',\n", " 'huggingface:nkjp-ner',\n", " 'huggingface:nli_tr',\n", " 'huggingface:nlu_evaluation_data',\n", " 'huggingface:norec',\n", " 'huggingface:norne',\n", " 'huggingface:norwegian_ner',\n", " 'huggingface:nq_open',\n", " 'huggingface:nsmc',\n", " 'huggingface:numer_sense',\n", " 'huggingface:numeric_fused_head',\n", " 'huggingface:oclar',\n", " 'huggingface:offcombr',\n", " 'huggingface:offenseval2020_tr',\n", " 'huggingface:offenseval_dravidian',\n", " 'huggingface:ofis_publik',\n", " 'huggingface:ohsumed',\n", " 'huggingface:ollie',\n", " 'huggingface:omp',\n", " 'huggingface:onestop_english',\n", " 'huggingface:onestop_qa',\n", " 'huggingface:open_subtitles',\n", " 'huggingface:openai_humaneval',\n", " 'huggingface:openbookqa',\n", " 'huggingface:openslr',\n", " 'huggingface:openwebtext',\n", " 'huggingface:opinosis',\n", " 'huggingface:opus100',\n", " 'huggingface:opus_books',\n", " 'huggingface:opus_dgt',\n", " 'huggingface:opus_dogc',\n", " 'huggingface:opus_elhuyar',\n", " 'huggingface:opus_euconst',\n", " 'huggingface:opus_finlex',\n", " 'huggingface:opus_fiskmo',\n", " 'huggingface:opus_gnome',\n", " 'huggingface:opus_infopankki',\n", " 'huggingface:opus_memat',\n", " 'huggingface:opus_montenegrinsubs',\n", " 'huggingface:opus_openoffice',\n", " 'huggingface:opus_paracrawl',\n", " 'huggingface:opus_rf',\n", " 'huggingface:opus_tedtalks',\n", " 'huggingface:opus_ubuntu',\n", " 'huggingface:opus_wikipedia',\n", " 'huggingface:opus_xhosanavy',\n", " 'huggingface:orange_sum',\n", " 'huggingface:oscar',\n", " 'huggingface:para_crawl',\n", " 'huggingface:para_pat',\n", " 'huggingface:parsinlu_reading_comprehension',\n", " 'huggingface:pass',\n", " 'huggingface:paws',\n", " 'huggingface:paws-x',\n", " 'huggingface:pec',\n", " 'huggingface:peer_read',\n", " 'huggingface:peoples_daily_ner',\n", " 'huggingface:per_sent',\n", " 'huggingface:persian_ner',\n", " 'huggingface:pg19',\n", " 'huggingface:php',\n", " 'huggingface:piaf',\n", " 'huggingface:pib',\n", " 'huggingface:piqa',\n", " 'huggingface:pn_summary',\n", " 'huggingface:poem_sentiment',\n", " 'huggingface:polemo2',\n", " 'huggingface:poleval2019_cyberbullying',\n", " 'huggingface:poleval2019_mt',\n", " 'huggingface:polsum',\n", " 'huggingface:polyglot_ner',\n", " 'huggingface:prachathai67k',\n", " 'huggingface:pragmeval',\n", " 'huggingface:proto_qa',\n", " 'huggingface:psc',\n", " 'huggingface:ptb_text_only',\n", " 'huggingface:pubmed',\n", " 'huggingface:pubmed_qa',\n", " 'huggingface:py_ast',\n", " 'huggingface:qa4mre',\n", " 'huggingface:qa_srl',\n", " 'huggingface:qa_zre',\n", " 'huggingface:qangaroo',\n", " 'huggingface:qanta',\n", " 'huggingface:qasc',\n", " 'huggingface:qasper',\n", " 'huggingface:qed',\n", " 'huggingface:qed_amara',\n", " 'huggingface:quac',\n", " 'huggingface:quail',\n", " 'huggingface:quarel',\n", " 'huggingface:quartz',\n", " 'huggingface:quickdraw',\n", " 'huggingface:quora',\n", " 'huggingface:quoref',\n", " 'huggingface:race',\n", " 'huggingface:re_dial',\n", " 'huggingface:reasoning_bg',\n", " 'huggingface:recipe_nlg',\n", " 'huggingface:reclor',\n", " 'huggingface:red_caps',\n", " 'huggingface:reddit',\n", " 'huggingface:reddit_tifu',\n", " 'huggingface:refresd',\n", " 'huggingface:reuters21578',\n", " 'huggingface:riddle_sense',\n", " 'huggingface:ro_sent',\n", " 'huggingface:ro_sts',\n", " 'huggingface:ro_sts_parallel',\n", " 'huggingface:roman_urdu',\n", " 'huggingface:roman_urdu_hate_speech',\n", " 'huggingface:ronec',\n", " 'huggingface:ropes',\n", " 'huggingface:rotten_tomatoes',\n", " 'huggingface:russian_super_glue',\n", " 'huggingface:rvl_cdip',\n", " 'huggingface:s2orc',\n", " 'huggingface:samsum',\n", " 'huggingface:sanskrit_classic',\n", " 'huggingface:saudinewsnet',\n", " 'huggingface:sberquad',\n", " 'huggingface:sbu_captions',\n", " 'huggingface:scan',\n", " 'huggingface:scb_mt_enth_2020',\n", " 'huggingface:scene_parse_150',\n", " 'huggingface:schema_guided_dstc8',\n", " 'huggingface:scicite',\n", " 'huggingface:scielo',\n", " 'huggingface:scientific_papers',\n", " 'huggingface:scifact',\n", " 'huggingface:sciq',\n", " 'huggingface:scitail',\n", " 'huggingface:scitldr',\n", " 'huggingface:search_qa',\n", " 'huggingface:sede',\n", " 'huggingface:selqa',\n", " 'huggingface:sem_eval_2010_task_8',\n", " 'huggingface:sem_eval_2014_task_1',\n", " 'huggingface:sem_eval_2018_task_1',\n", " 'huggingface:sem_eval_2020_task_11',\n", " 'huggingface:sent_comp',\n", " 'huggingface:senti_lex',\n", " 'huggingface:senti_ws',\n", " 'huggingface:sentiment140',\n", " 'huggingface:sepedi_ner',\n", " 'huggingface:sesotho_ner_corpus',\n", " 'huggingface:setimes',\n", " 'huggingface:setswana_ner_corpus',\n", " 'huggingface:sharc',\n", " 'huggingface:sharc_modified',\n", " 'huggingface:sick',\n", " 'huggingface:silicone',\n", " 'huggingface:simple_questions_v2',\n", " 'huggingface:siswati_ner_corpus',\n", " 'huggingface:smartdata',\n", " 'huggingface:sms_spam',\n", " 'huggingface:snips_built_in_intents',\n", " 'huggingface:snli',\n", " 'huggingface:snow_simplified_japanese_corpus',\n", " 'huggingface:so_stacksample',\n", " 'huggingface:social_bias_frames',\n", " 'huggingface:social_i_qa',\n", " 'huggingface:sofc_materials_articles',\n", " ...]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tfds.list_builders()" ] }, { "cell_type": "markdown", "metadata": { "id": "VjI6VgOBf0v0" }, "source": [ "## Load a dataset\n", "\n", "### tfds.load\n", "\n", "The easiest way of loading a dataset is `tfds.load`. It will:\n", "\n", "1. Download the data and save it as [`tfrecord`](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.\n", "2. Load the `tfrecord` and create the `tf.data.Dataset`.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:41:58.192650Z", "iopub.status.busy": "2024-12-14T12:41:58.192104Z", "iopub.status.idle": "2024-12-14T12:41:58.941585Z", "shell.execute_reply": "2024-12-14T12:41:58.940719Z" }, "id": "dCou80mnLLPV" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "<_PrefetchDataset element_spec={'image': TensorSpec(shape=(28, 28, 1), dtype=tf.uint8, name=None), 'label': TensorSpec(shape=(), dtype=tf.int64, name=None)}>\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2024-12-14 12:41:58.857496: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n" ] } ], "source": [ "ds = tfds.load('mnist', split='train', shuffle_files=True)\n", "assert isinstance(ds, tf.data.Dataset)\n", "print(ds)" ] }, { "cell_type": "markdown", "metadata": { "id": "byOXYCEJS7S6" }, "source": [ "Some common arguments:\n", "\n", "* `split=`: Which split to read (e.g. `'train'`, `['train', 'test']`, `'train[80%:]'`,...). See our [split API guide](https://www.tensorflow.org/datasets/splits).\n", "* `shuffle_files=`: Control whether to shuffle the files between each epoch (TFDS store big datasets in multiple smaller files).\n", "* `data_dir=`: Location where the dataset is saved (\n", "defaults to `~/tensorflow_datasets/`)\n", "* `with_info=True`: Returns the `tfds.core.DatasetInfo` containing dataset metadata\n", "* `download=False`: Disable download\n" ] }, { "cell_type": "markdown", "metadata": { "id": "qeNmFx_1RXCb" }, "source": [ "### tfds.builder\n", "\n", "`tfds.load` is a thin wrapper around `tfds.core.DatasetBuilder`. You can get the same output using the `tfds.core.DatasetBuilder` API:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:41:58.944739Z", "iopub.status.busy": "2024-12-14T12:41:58.944448Z", "iopub.status.idle": "2024-12-14T12:41:59.393465Z", "shell.execute_reply": "2024-12-14T12:41:59.392688Z" }, "id": "2zN_jQ2ER40W" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "<_PrefetchDataset element_spec={'image': TensorSpec(shape=(28, 28, 1), dtype=tf.uint8, name=None), 'label': TensorSpec(shape=(), dtype=tf.int64, name=None)}>\n" ] } ], "source": [ "builder = tfds.builder('mnist')\n", "# 1. Create the tfrecord files (no-op if already exists)\n", "builder.download_and_prepare()\n", "# 2. Load the `tf.data.Dataset`\n", "ds = builder.as_dataset(split='train', shuffle_files=True)\n", "print(ds)" ] }, { "cell_type": "markdown", "metadata": { "id": "IwrjccfjoQCD" }, "source": [ "### `tfds build` CLI\n", "\n", "If you want to generate a specific dataset, you can use the [`tfds` command line](https://www.tensorflow.org/datasets/cli). For example:\n", "\n", "```sh\n", "tfds build mnist\n", "```\n", "\n", "See [the doc](https://www.tensorflow.org/datasets/cli) for available flags." ] }, { "cell_type": "markdown", "metadata": { "id": "aW132I-rbJXE" }, "source": [ "## Iterate over a dataset\n", "\n", "### As dict\n", "\n", "By default, the `tf.data.Dataset` object contains a `dict` of `tf.Tensor`s:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:41:59.396781Z", "iopub.status.busy": "2024-12-14T12:41:59.396222Z", "iopub.status.idle": "2024-12-14T12:42:00.187994Z", "shell.execute_reply": "2024-12-14T12:42:00.187195Z" }, "id": "JAGjXdk_bIYQ" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['image', 'label']\n", "(28, 28, 1) tf.Tensor(4, shape=(), dtype=int64)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2024-12-14 12:42:00.177868: W tensorflow/core/kernels/data/cache_dataset_ops.cc:914] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset will be discarded. This can happen if you have an input pipeline similar to `dataset.cache().take(k).repeat()`. You should use `dataset.take(k).cache().repeat()` instead.\n" ] } ], "source": [ "ds = tfds.load('mnist', split='train')\n", "ds = ds.take(1) # Only take a single example\n", "\n", "for example in ds: # example is `{'image': tf.Tensor, 'label': tf.Tensor}`\n", " print(list(example.keys()))\n", " image = example[\"image\"]\n", " label = example[\"label\"]\n", " print(image.shape, label)" ] }, { "cell_type": "markdown", "metadata": { "id": "sIqX2bmhu-8d" }, "source": [ "To find out the `dict` key names and structure, look at the dataset documentation in [our catalog](https://www.tensorflow.org/datasets/catalog/overview#all_datasets). For example: [mnist documentation](https://www.tensorflow.org/datasets/catalog/mnist)." ] }, { "cell_type": "markdown", "metadata": { "id": "umAtqBBqdkDG" }, "source": [ "### As tuple (`as_supervised=True`)\n", "\n", "By using `as_supervised=True`, you can get a tuple `(features, label)` instead for supervised datasets." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:00.191284Z", "iopub.status.busy": "2024-12-14T12:42:00.190607Z", "iopub.status.idle": "2024-12-14T12:42:01.235242Z", "shell.execute_reply": "2024-12-14T12:42:01.234402Z" }, "id": "nJ4O0xy3djfV" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(28, 28, 1) tf.Tensor(4, shape=(), dtype=int64)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2024-12-14 12:42:01.224313: W tensorflow/core/kernels/data/cache_dataset_ops.cc:914] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset will be discarded. This can happen if you have an input pipeline similar to `dataset.cache().take(k).repeat()`. You should use `dataset.take(k).cache().repeat()` instead.\n" ] } ], "source": [ "ds = tfds.load('mnist', split='train', as_supervised=True)\n", "ds = ds.take(1)\n", "\n", "for image, label in ds: # example is (image, label)\n", " print(image.shape, label)" ] }, { "cell_type": "markdown", "metadata": { "id": "u9palgyHfEwQ" }, "source": [ "### As numpy (`tfds.as_numpy`)\n", "\n", "Uses `tfds.as_numpy` to convert:\n", "\n", "* `tf.Tensor` -> `np.array`\n", "* `tf.data.Dataset` -> `Iterator[Tree[np.array]]` (`Tree` can be arbitrary nested `Dict`, `Tuple`)\n", "\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:01.238403Z", "iopub.status.busy": "2024-12-14T12:42:01.237749Z", "iopub.status.idle": "2024-12-14T12:42:02.076283Z", "shell.execute_reply": "2024-12-14T12:42:02.075485Z" }, "id": "tzQTCUkAfe9R" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 4\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2024-12-14 12:42:02.065492: W tensorflow/core/kernels/data/cache_dataset_ops.cc:914] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset will be discarded. This can happen if you have an input pipeline similar to `dataset.cache().take(k).repeat()`. You should use `dataset.take(k).cache().repeat()` instead.\n" ] } ], "source": [ "ds = tfds.load('mnist', split='train', as_supervised=True)\n", "ds = ds.take(1)\n", "\n", "for image, label in tfds.as_numpy(ds):\n", " print(type(image), type(label), label)" ] }, { "cell_type": "markdown", "metadata": { "id": "XaRN-LdXUkl_" }, "source": [ "### As batched tf.Tensor (`batch_size=-1`)\n", "\n", "By using `batch_size=-1`, you can load the full dataset in a single batch.\n", "\n", "This can be combined with `as_supervised=True` and `tfds.as_numpy` to get the the data as `(np.array, np.array)`:\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:02.079446Z", "iopub.status.busy": "2024-12-14T12:42:02.078810Z", "iopub.status.idle": "2024-12-14T12:42:02.853462Z", "shell.execute_reply": "2024-12-14T12:42:02.852673Z" }, "id": "Gg8BNsv-UzFl" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " (10000, 28, 28, 1)\n" ] } ], "source": [ "image, label = tfds.as_numpy(tfds.load(\n", " 'mnist',\n", " split='test',\n", " batch_size=-1,\n", " as_supervised=True,\n", "))\n", "\n", "print(type(image), image.shape)" ] }, { "cell_type": "markdown", "metadata": { "id": "kRJrB3L6wgKI" }, "source": [ "Be careful that your dataset can fit in memory, and that all examples have the same shape." ] }, { "cell_type": "markdown", "metadata": { "id": "heaKNg7-X4jN" }, "source": [ "## Benchmark your datasets\n", "\n", "Benchmarking a dataset is a simple `tfds.benchmark` call on any iterable (e.g. `tf.data.Dataset`, `tfds.as_numpy`,...).\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:02.856937Z", "iopub.status.busy": "2024-12-14T12:42:02.856283Z", "iopub.status.idle": "2024-12-14T12:42:04.922912Z", "shell.execute_reply": "2024-12-14T12:42:04.922187Z" }, "id": "ZyQzZ98bX3dM" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "************ Summary ************\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Examples/sec (First included) 45061.33 ex/sec (total: 60032 ex, 1.33 sec)\n", "Examples/sec (First only) 72.23 ex/sec (total: 32 ex, 0.44 sec)\n", "Examples/sec (First excluded) 67477.64 ex/sec (total: 60000 ex, 0.89 sec)\n", "\n", "************ Summary ************\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Examples/sec (First included) 199832.93 ex/sec (total: 60032 ex, 0.30 sec)\n", "Examples/sec (First only) 2367.04 ex/sec (total: 32 ex, 0.01 sec)\n", "Examples/sec (First excluded) 209137.96 ex/sec (total: 60000 ex, 0.29 sec)\n" ] }, { "data": { "text/html": [ "BenchmarkResult:
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
durationnum_examplesavg
first+lasts0.30041160032199832.925534
first0.013519322367.040208
lasts0.28689260000209137.961140
\n", "
" ], "text/plain": [ "BenchmarkResult(stats= duration num_examples avg\n", "first+lasts 0.300411 60032 199832.925534\n", "first 0.013519 32 2367.040208\n", "lasts 0.286892 60000 209137.961140, raw_stats= duration\n", "start_time 2465.173582\n", "first_batch_time 2465.187101\n", "end_time 2465.473993\n", "num_iter 1875.000000)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = tfds.load('mnist', split='train')\n", "ds = ds.batch(32).prefetch(1)\n", "\n", "tfds.benchmark(ds, batch_size=32)\n", "tfds.benchmark(ds, batch_size=32) # Second epoch much faster due to auto-caching" ] }, { "cell_type": "markdown", "metadata": { "id": "MT0yEX_4kYnV" }, "source": [ "* Do not forget to normalize the results per batch size with the `batch_size=` kwarg.\n", "* In the summary, the first warmup batch is separated from the other ones to capture `tf.data.Dataset` extra setup time (e.g. buffers initialization,...).\n", "* Notice how the second iteration is much faster due to [TFDS auto-caching](https://www.tensorflow.org/datasets/performances#auto-caching).\n", "* `tfds.benchmark` returns a `tfds.core.BenchmarkResult` which can be inspected for further analysis." ] }, { "cell_type": "markdown", "metadata": { "id": "o-cuwvVbeb43" }, "source": [ "### Build end-to-end pipeline\n", "\n", "To go further, you can look:\n", "\n", "* Our [end-to-end Keras example](https://www.tensorflow.org/datasets/keras_example) to see a full training pipeline (with batching, shuffling,...).\n", "* Our [performance guide](https://www.tensorflow.org/datasets/performances) to improve the speed of your pipelines (tip: use `tfds.benchmark(ds)` to benchmark your datasets).\n" ] }, { "cell_type": "markdown", "metadata": { "id": "gTRTEQqscxAE" }, "source": [ "## Visualization\n", "\n", "### tfds.as_dataframe\n", "\n", "`tf.data.Dataset` objects can be converted to [`pandas.DataFrame`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) with `tfds.as_dataframe` to be visualized on [Colab](https://colab.research.google.com).\n", "\n", "* Add the `tfds.core.DatasetInfo` as second argument of `tfds.as_dataframe` to visualize images, audio, texts, videos,...\n", "* Use `ds.take(x)` to only display the first `x` examples. `pandas.DataFrame` will load the full dataset in-memory, and can be very expensive to display." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:04.926139Z", "iopub.status.busy": "2024-12-14T12:42:04.925607Z", "iopub.status.idle": "2024-12-14T12:42:06.157855Z", "shell.execute_reply": "2024-12-14T12:42:06.157137Z" }, "id": "FKouwN_yVSGQ" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-12-14 12:42:05.775782: W tensorflow/core/kernels/data/cache_dataset_ops.cc:914] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset will be discarded. This can happen if you have an input pipeline similar to `dataset.cache().take(k).repeat()`. You should use `dataset.take(k).cache().repeat()` instead.\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 imagelabel
0\"Img\"4
1\"Img\"1
2\"Img\"0
3\"Img\"7
\n" ], "text/plain": [ " image label\n", "0 [[[0], [0], [0], [0], [0], [0], [0], [0], [0],... 4\n", "1 [[[0], [0], [0], [0], [0], [0], [0], [0], [0],... 1\n", "2 [[[0], [0], [0], [0], [0], [0], [0], [0], [0],... 0\n", "3 [[[0], [0], [0], [0], [0], [0], [0], [0], [0],... 7" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds, info = tfds.load('mnist', split='train', with_info=True)\n", "\n", "tfds.as_dataframe(ds.take(4), info)" ] }, { "cell_type": "markdown", "metadata": { "id": "b-eDO_EXVGWC" }, "source": [ "### tfds.show_examples\n", "\n", "`tfds.show_examples` returns a `matplotlib.figure.Figure` (only image datasets supported now):" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:06.161149Z", "iopub.status.busy": "2024-12-14T12:42:06.160454Z", "iopub.status.idle": "2024-12-14T12:42:07.144860Z", "shell.execute_reply": "2024-12-14T12:42:07.144136Z" }, "id": "DpE2FD56cSQR" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-12-14 12:42:06.859043: W tensorflow/core/kernels/data/cache_dataset_ops.cc:914] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset will be discarded. This can happen if you have an input pipeline similar to `dataset.cache().take(k).repeat()`. You should use `dataset.take(k).cache().repeat()` instead.\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAswAAALcCAYAAADzB+aBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8ekN5oAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/CElEQVR4nO3dfZiVVaE3/rV5Bx1E5MUmRRECTEFG0YIMMd8g8zxplqAnEDVN8aTHx3y08yhlxwzN7BD4QnKw54SVp4QoScOXFE0L0VGQox0IEIFREZBBxFHm/v1xrvhF7LXYs2fPwLA/n+vij9Z3r/te2/ZivtzMLHJZlmUBAADIq9XuXgAAAOzJFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIKFNIS+qr68Pa9asCRUVFSGXyzX1mmC3y7Is1NbWhsrKytCqVXn+udK+p9yU+7635yk3DdnzBRXmNWvWhIMPPrgki4OWZNWqVeGggw7a3cvYLex7ylW57nt7nnJVyJ4v6I/QFRUVJVkQtDTl/Nkv5/dOeSvXz365vm8o5LNfUGH2VzOUq3L+7Jfze6e8letnv1zfNxTy2S+/b9ICAIAGUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEtrs7gUAABDXr1+/aDZ16tS84yeddFJ0zr333hvNLrvssmi2devWaLa384QZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhwrBwAwB5s2LBh0ewzn/lM3vEsy6Jzxo0bF822bdsWzSZMmJB3vK6uLjpnb+EJMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJCjMAACQ4Fi5MnT22WdHs/vvvz+aXXLJJdHsRz/6UaPWBJRGx44d847fcccd0TmdOnWKZmPGjIlm9fX1hS8MSBo5cmQ0+8EPftBs67jgggui2ZIlS/KO33777U21nD2GJ8wAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQIJj5crQueeeG82yLItmXbt2bYrlAA2Uy+Wi2d133513/B//8R+LutfNN98czaqrq4u6JpSr1BGON954YzSrqKhoiuU02PXXX5933LFyAABQ5hRmAABIUJgBACBBYQYAgASFGQAAEpySsRc75JBD8o6PGjUqOmfhwoXR7L777mv0moDG+/jHPx7NijkNY9OmTdHs7bffbvD1gPx++ctfRrMhQ4ZEs9QJVjGpU2wGDx7c4OuFEEKbNuVbGz1hBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASyvd8kALkcrmi5hVz/EtT+NrXvpZ3vF27dtE5f/nLX6LZqlWrGr0moPG++MUvlvR6r732WjSz76FhLrroomg2YsSIkt8v9nX7hBNOiM5JHW938sknR7PYsXJ9+vSJzlm2bFk0a0k8YQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhwrl5A6/uX222+PZl/96lfzjj/77LONXVKDDBw4sMFzqqurS78QoKSuuOKKBs/58MMPo9nNN9/cmOVAWRo7dmze8SlTpkTntG3btqh7LV26NJqddtppecc3b94cnfP2228XtY727dvnHU/1JcfKAQBAGVCYAQAgQWEGAIAEhRkAABIUZgAASFCYAQAgwbFyCe+99140Sx3ZdsIJJ+Qdb4pj5Q466KAGr6O2tjY658c//nGj1wQ0XpcuXaLZfvvt1+DrvfXWW9Hspz/9aYOvB+Xgox/9aDS77rrr8o4Xe3Tc2rVro9kll1wSzVasWFHU/UrppJNOimbTp09vxpU0HU+YAQAgQWEGAIAEhRkAABIUZgAASFCYAQAgQWEGAIAEx8olvPnmm7t7Cbt05plnRrPY0TbPPfdcdE7qWBug+dx4440lvd6iRYtKej3YW6SOZ507d24069evX0nXccstt0Sz3//+9yW9V6kdccQRu3sJTc4TZgAASFCYAQAgQWEGAIAEhRkAABIUZgAASHBKRkLXrl139xJ2qbKyssFz9vSftgVCuOiii0p6vX/7t38r6fVgbzF9+vRoVurTH6qrq6PZvffeW9J7NaeWvPZCecIMAAAJCjMAACQozAAAkKAwAwBAgsIMAAAJCjMAACQ4Vi7hzDPPjGa5XK7Z1vHRj340ml166aXRLLbGf//3f2/0moA908aNG/OOz5s3r3kXAnuQ0047LZqdcsopJb3Xu+++G80+//nPR7N33nmnpOtISXWYYvpNbW1tY5bTInjCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAklP2xcu3bt49mF198cTTLsiyajRkzJu/4oYceGp3TtWvXaDZo0KBoVlFREc1eeOGFvOPLly+PzgGaz+DBg6NZ27Zti7rm1KlT845/+OGHRV0PWoouXbpEs3vuuSeapb6ep8SOjxs3blx0zqpVq4q6VzHatWsXzXr06BHNUv89tm3blnd89erVhS+shfKEGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIKPtj5c4999xoljrqLWXgwIF5x1PHwxV7rE3Kd7/73bzj9fX1Jb8X0HC33HJLNGvTJv7b8wcffBDNYsfKwd4udUxsZWVlye/361//Ou/4rFmzSn6vYvzTP/1TNBsxYkRR19y6dWve8d/+9rdFXa8l8YQZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEgo+2Pljj322Gi2ZcuWaPbv//7v0WzNmjV5x9evXx+ds27dumj2i1/8IpqlPPTQQ0XNA0rnkEMOiWZDhw6NZqmjJpcuXRrNampqClsYtFDDhw/POz5nzpyS3yu1D+fOnVvy+5XS5z73uZJfs127dnnHhwwZEp3z3HPPlXwdu4MnzAAAkKAwAwBAgsIMAAAJCjMAACQozAAAkFD2p2RcdtllRWWldvbZZ0ezXC4XzR544IFotmnTpkatCWi8q6++Oprts88+RV3zlltuKXY50OJNmTIl73hFRUXJ7/WXv/wlms2cObPk9yvGiSeemHf8U5/6VMnvVV9fn3d8w4YNJb/XnsYTZgAASFCYAQAgQWEGAIAEhRkAABIUZgAASFCYAQAgoeyPldtTnHvuudEsy7JotmDBgqZYDlAiI0aMKPk177333pJfE1qK+++/P+/4t771rZLf6+c//3nJr1mMf/zHf4xm3/zmN/OOt27duuTrmDhxYt7xZcuWlfxeexpPmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABMfK7SFOOOGEaJY6Vu6JJ55oiuUADXTUUUflHe/Xr19R15s9e3YjVgN7r5qamma7V7t27aLZhRdemHf8mGOOic5ZtWpVNEsdQTl8+PBollpjTH19fTSLHdsXQgi33XZbg++1t/CEGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIcKxcMzr66KOjWZs28f8rfve730WzZ599tlFrAkpjypQpecfbtm1b1PVuvPHGxiwHKIGrr7662e7VqlX8GWbqGLiYN954I5p9//vfj2bf+973GnyvcuAJMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJDgloxlNmjQpmlVUVESzk046KZpdeuml0ezOO+8sbGFAQfbdd99odthhhzX4ehs2bIhmS5YsafD1oBzMnTs373hqz3z84x9vquWUTJZl0WzdunXRbNq0aXnHp0+fHp2zYsWKgtfF//CEGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIcKxcM0odGZPKXn755Wj2i1/8olFrAgrXr1+/aPaRj3ykwdf7wx/+EM3q6uoafD0oB2vWrMk7Pnz48Oic0aNHR7Prr78+mvXs2bPwhRXg3nvvjWa/+c1votkzzzwTzWpqahqzJArkCTMAACQozAAAkKAwAwBAgsIMAAAJCjMAACQozAAAkOBYuWZ0+OGHR7N33303mp111lnR7K233mrUmoDCnXHGGSW93j333FPS60E527BhQzS78847i8rgrzxhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASHCvXjDp27BjN3njjjWi2YsWKJlgN0FBTp06NZpdddlne8SzLonMeeeSRRq8JgKbnCTMAACQozAAAkKAwAwBAgsIMAAAJCjMAACQozAAAkOBYuWbUrVu33b0EoBHWrVsXzXr27NmMKwGgOXnCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkFFSYsyxr6nXAHqmcP/vl/N4pb+X62S/X9w2FfPYLKsy1tbWNXgy0ROX82S/n9055K9fPfrm+byjks5/LCqjV9fX1Yc2aNaGioiLkcrmSLA72ZFmWhdra2lBZWRlatSrP71yy7yk35b7v7XnKTUP2fEGFGQAAylX5/REaAAAaQGEGAIAEhRkAABIUZgAASFCY92B1dXWhb9++4Q9/+EPBcx566KEwePDgUF9f34QrA5rS8OHDw3333Vfw69etWxd69OgRXn/99SZcFdAUfK1vGRTmZvbd73435HK5cOWVV+7ytXfddVfo3bt3GDZs2E7Z+++/HwYPHhxyuVyorq7ePj5y5MjQtm3bMHPmzBKuGmioJ598MpxxxhmhsrIy5HK5MHv27ILmzZkzJ7zxxhth9OjR28emTZsWRowYETp37hxyuVzYuHHjDnO6desWxo4dGyZOnFjCdwA01NSpU8Ohhx4aOnToED7xiU+EP/3pT7uck+9r/fr168N5550XOnfuHLp06RIuvPDCsHnz5u25r/XNT2FuRgsWLAh33313GDRo0C5fm2VZmDJlSrjwwgvz5tdcc02orKzMm51//vlh8uTJjVor0DjvvvtuOOqoo8LUqVMbNG/y5Mlh/PjxO5wJumXLljBy5MjwjW98Izpv/PjxYebMmWH9+vVFrxko3s9//vNw1VVXhYkTJ4bnn38+HHXUUeG0004Lb775ZnRO7Gv9eeedF15++eUwb9688Jvf/CY8+eST4eKLL97hNb7WN7OMZlFbW5t97GMfy+bNm5edcMIJ2RVXXJF8/YIFC7JWrVplmzZt2imbO3duNmDAgOzll1/OQgjZCy+8sEO+cuXKLISQLV26tITvAChWCCGbNWvWLl/35ptvZrlcLlu8eHHe/PHHH89CCNmGDRvy5r17987uueeeRqwUKNZxxx2XTZgwYfv/3rZtW1ZZWZndfPPN0Tn5vtYvWbIkCyFkCxYs2D7229/+Nsvlctnq1au3j/la37w8YW4mEyZMCKeffno4+eSTC3r9/PnzQ79+/UJFRcUO42+88Ub4yle+Ev7jP/4jdOrUKe/cXr16hZ49e4b58+c3et1A83nqqadCp06dwuGHH17U/OOOO86+h92grq4uLFy4cIev8a1atQonn3xyeOaZZ6Lz8n2tf+aZZ0KXLl3CkCFDto+dfPLJoVWrVuGPf/zj9jFf65tXm929gHLws5/9LDz//PNhwYIFBc9ZuXLlTt9ykWVZOP/888NXv/rVMGTIkLBixYro/MrKyrBy5cpilwzsBitXrgw9e/Ys+p9lrqysDC+88EKJVwXsyrp168K2bdtCz549dxjv2bNneOWVV6Lz8n2tr6mpCT169NhhrE2bNqFr166hpqZmh3Ff65uPwtzEVq1aFa644oowb9680KFDh4Lnvffeezu9/oc//GGora0N11133S7nd+zYMWzZsqXB6wV2n3z7viHse2hZ7PmWw7dkNLGFCxeGN998Mxx99NGhTZs2oU2bNuGJJ54IkydPDm3atAnbtm3LO69bt25hw4YNO4w99thj4Zlnngnt27cPbdq0CX379g0hhDBkyJAwbty4HV67fv360L1796Z5U0CTyLfvG8K+h92jW7duoXXr1uGNN97YYfyNN94IBx54YHLe3+/5Aw88cKcfFPzwww/D+vXrd7qWPd98FOYmdtJJJ4VFixaF6urq7b+GDBkSzjvvvFBdXR1at26dd15VVVV45ZVXQpZl28cmT54cXnzxxe3XmTt3bgjhf34y96abbtr+uq1bt4Zly5aFqqqqpn1zQElVVVWFmpqaokvz4sWL7XvYDdq1axeOOeaY8Oijj24fq6+vD48++mgYOnRodF6+r/VDhw4NGzduDAsXLtw+9thjj4X6+vrwiU98YvuYr/XNy7dkNLGKiopw5JFH7jC2zz77hAMOOGCn8b914oknhs2bN4eXX355++t69eq1w2v23XffEEIIffr0CQcddND28WeffTa0b98+uUmBprV58+awdOnS7f97+fLlobq6OnTt2nWnvfxXVVVVoVu3buHpp58On/vc57aP19TUhJqamu3XW7RoUaioqAi9evUKXbt2DSH8z9FzCxcuDN/5znea8F0BMVdddVUYN25cGDJkSDjuuOPCD37wg/Duu++G8ePHR+fk+1p/+OGHh5EjR4avfOUr4a677goffPBBuPzyy8Po0aN3+H5nX+ublyfMe6gDDjggnHnmmUUdSv7Tn/40nHfeedFTNICm99xzz4WqqqrtT3+uuuqqUFVVFW644YbonNatW28/T/lv3XXXXaGqqip85StfCSH8z78EWFVVFebMmbP9Nb/61a9Cr169wqc//ekmeDfArpxzzjnhe9/7XrjhhhvC4MGDQ3V1dXjooYd2+kHAvxX7Wj9z5swwYMCAcNJJJ4XPfvaz4fjjjw/Tpk3b4TW+1jevXPa3fw/AHuWll14Kp5xySli2bNn2p8m7sm7dutC/f//w3HPPhd69ezfxCoFSq6mpCUcccUR4/vnnwyGHHFLwvE9+8pPha1/7Wjj33HObcHVAqfla3zJ4wrwHGzRoUJg0aVJYvnx5wXNWrFgR7rjjDhsIWqgDDzwwTJ8+Pbz22msFz1m3bl0466yzwpgxY5pwZUBT8LW+ZfCEGQAAEjxhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCAhDaFvKi+vj6sWbMmVFRUhFwu19Rrgt0uy7JQW1sbKisrQ6tW5fnnSvueclPu+96ep9w0ZM8XVJjXrFkTDj744JIsDlqSVatWhYMOOmh3L2O3sO8pV+W67+15ylUhe76gP0JXVFSUZEHQ0pTzZ7+c3zvlrVw/++X6vqGQz35BhdlfzVCuyvmzX87vnfJWrp/9cn3fUMhnv/y+SQsAABpAYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEtrs7gUA0PQqKiqi2YQJE6LZd77znWi2du3avOMf//jHo3PeeeedaAbk1759+2j29NNP5x0/7LDDonNOPvnkaPb8888XvrAy4gkzAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJDgWDmAEogd4ZQ6zu0LX/hCNOvQoUOD75XKXnzxxeicsWPHRrMsy6LZRz7ykbzjqbU7Vg4abv/9949mRx99dIOvd++990azY489Npq9//77Db7X3sITZgAASFCYAQAgQWEGAIAEhRkAABIUZgAASFCYAQAgwbFyCY8//ng0GzFiRDSbNGlS3vFrr722sUsCSqB9+/bRrHfv3tHszjvvjGZVVVV5xzt37hydkzqyrVi5XC7v+FFHHVXyewHN45vf/GZJr5f6fal79+7R7PXXXy/pOloST5gBACBBYQYAgASFGQAAEhRmAABIUJgBACChLE7JiP3UeAgh9O/fP5rFfuo9hBDq6+uj2RVXXJF3fNu2bdE5DzzwQDRL/ST9q6++Gs1iPvOZz0Szww47LJqtWLEims2dOzfv+AcffFDwuqCUUp/l+++/P5ql9n0xnn766Wi2bNmyaPbggw9Gs40bN0azhx9+uKB1lcLq1avzjm/durXZ1gB7izPPPDOaXXLJJdGsmNN2lixZEs3K+SSMFE+YAQAgQWEGAIAEhRkAABIUZgAASFCYAQAgQWEGAICEsjhWbuDAgdHshRdeKPn92rVrl3f82muvjc5JZS3B/Pnz846njsnZsGFDUy2HMjJq1Ki846lj2VJqa2uj2eOPPx7Nbr311rzjqWPlivXlL3+5wXM2b95c1L0qKiqi2aOPPpp3/J133inqXlDOBgwYUNLrxY59DCGECy64oKT3KgeeMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACXvVsXKHHHJI3vHZs2eX/F6bNm2KZvX19XnH999//+icLMuKWkculyvpNVPHQe23337RbPjw4XnHb7rppuicyy67rPCFUdaOOOKIaBbb36nP/5/+9KdodvbZZ0ez1DFNzWnhwoXRbOrUqXnHX3/99eicf/7nf45m++67bzS79NJLoxnQMGPHji3p9aZNmxbNampqSnqvcuAJMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJCjMAACQsFcdK3fxxRfnHY8dN7crkyZNimY/+MEPotl7772Xd/wzn/lMUetoTosXL45mf/7znxt8vYqKisYsB0IIIQwaNCiatWnT8N/GPvvZz0azDRs2NPh6zW3JkiXR7J/+6Z/yjo8ZMyY6p3v37tFsy5Yt0Sz2ex2QX2offuxjHyvpvVatWlXS65U7T5gBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgIQWd6zc8ccfH82uvPLKkt5r8uTJ0ezNN99s8PV+9atfNWY5zaJv375FzcuyLO/4aaedFp3ToUOHaLZ169ai1sHeqaqqqqTXO+aYY6LZI488UtJ77Sm+/vWvFzXvtttuK/FKoHxdf/310axVq+KeYb711lt5xx944IGirkd+njADAECCwgwAAAkKMwAAJCjMAACQoDADAEBCizslI3U6RezUhbq6uuicKVOmRLMNGzYUvrC9xLnnnlvUvFwul3f84Ycfjs5xEgaFmjlzZjS7+uqrG3y93/3ud0Wt4ze/+U00i/1+sXbt2uic2bNnR7Nnn3224HX9rXHjxuUdHzx4cHROTU1NNPvmN79Z1DqAne2///4lv+btt9+ed3zTpk0lv1c584QZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhoccfK/fd//3c0O+KII/KO19bWRuesXr260Wvam3Tu3LmoeVmWlXgl8P9bsmRJNDv99NPzjt90003ROanPee/evRt8r5TYkYshhPDP//zP0eztt99u8L1CCGG//fbLO57ao6+99lo0O+qoo6LZiy++WPjCoEx8+ctfjmY9evQo6pqbN2+OZrfddltR16RhPGEGAIAEhRkAABIUZgAASFCYAQAgQWEGAIAEhRkAABJa3LFyqaORXnnllWZcSct14403RrMJEyYUdc3Y0X3Tp08v6nrwtz744INo9tvf/rZB4yGEUFFREc2KPVauS5cuecdTx8qlfj8bN25cNOvevXs0i90vda9jjz02mj3//PPRbNGiRXnHv/71r0fnzJs3L5rB3uCUU06JZq1aFfec8sMPP4xmqd8fKR1PmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCAhBZ3rByF+/a3v513/LrrrovOSR2BlXLPPffkHf/9739f1PWgKcWOQQwhhJdeeqmorBgnn3xyNLvkkkuKuubChQvzjt96663ROZ/97Gej2UknnRTNBg0alHf8P//zP6Nzjj766Gj2l7/8JZrBnmbw4MF5x88444zonNTxjim33HJLUfMoHU+YAQAgQWEGAIAEhRkAABIUZgAASFCYAQAgQWEGAIAEx8q1AKmj3s4777xo9r//9/9u8PVSHnvssWh27bXXFnVN2Nt985vfjGZf//rXo1nHjh2j2dNPPx3Nxo0bl3c8dWTb/fffH82OP/74aPbkk0/mHe/cuXN0zr777hvNoCX52Mc+lnd8v/32K/m9HnzwwZJfk4bxhBkAABIUZgAASFCYAQAgQWEGAIAEhRkAABKckrGHOPTQQ6PZt771rWj25S9/OZplWdbgdbz66qvRbPz48dHsww8/bPC9oKVp27ZtNJs9e3be8VGjRkXnpPbozJkzo9nll18ezd55551oVoyjjz66wXMWL14czZYsWdKY5UBZ+tSnPhXNXnrppWZcSfnyhBkAABIUZgAASFCYAQAgQWEGAIAEhRkAABIUZgAASHCsXDM68sgjo9mkSZOi2ciRI6NZMUfHzZo1K5pdffXV0ez1119v8L1gT3TggQdGs7PPPjuanXPOOQ2+5vvvvx+dk9r3qey9996LZsXYZ599otmll17a4OvdfPPN0cwRlOwtRo8e3Wz3uuWWW6LZnXfe2WzrKGeeMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACY6VawIf/ehH845Pnz49OmfIkCElX8fll1+ed9wRNOxNOnbsmHf8jjvuiM4ZN25cNCvmqMYQQnjkkUfyjl933XXROb/4xS+KulepDRw4MJr169cvmq1evTrv+Ny5cxu9JtjTHXbYYbt7CTQjT5gBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgATHyjWBK664Iu/4scceG52TOspq8+bN0ezaa6+NZvfcc080g5bkE5/4RDSbMmVK3vFjjjkmOieXy0Wz73//+9HspptuimYbNmyIZnuCXr16RbMHH3wwmqX+W33729/OO/7OO+8UvjBgl2bNmrW7l1D2PGEGAIAEhRkAABIUZgAASFCYAQAgQWEGAIAEp2QUKfbT4SHET8lInYSR+qny6667Lprdfffd0Qz2Fl/4whei2dFHH513PLXfUv7rv/4rmlVUVESz1CkUzWnYsGF5x1O/j3Tp0iWaLVu2LJpNmzat4HVBS3TCCSdEs8MPP7yk93rppZei2dixY0t6LxrOE2YAAEhQmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIMGxcgmpo5bOPffcaNamTf7/rLlcLjrnZz/7WTRzdBzl7t57741mZ5xxRt7xfv36FXWv1FFpGzZsiGb7779/3vHUvi/26LuU2P3q6uqic+bOnRvNUr/Xwd6uU6dO0axdu3YlvdeDDz5Y0utRWp4wAwBAgsIMAAAJCjMAACQozAAAkKAwAwBAgsIMAAAJjpVLGDNmTDQ79NBDG3y9v/zlL9HsO9/5ToOvB+ViyZIl0Wzw4MF5x4cPHx6d86lPfSqapfZ2x44do9nZZ58dzYqRes8LFy6MZjU1NXnHZ8+eHZ3z7LPPFrwuKCfz5s2LZldeeWXe8VNOOSU6Z9myZdHsiSeeKHhdND9PmAEAIEFhBgCABIUZAAASFGYAAEhQmAEAIEFhBgCAhFyWZdmuXrRp06aw3377Ncd69iijRo2KZg8++GA0i/0nvfTSS6Nzpk2bVvjCaDbvvPNO6Ny58+5exm5RrvseynXf2/OUq0L2vCfMAACQoDADAECCwgwAAAkKMwAAJCjMAACQoDADAEBCm929gD3ZY489Fs3++Mc/RrP+/fs3+HoAAOyZPGEGAIAEhRkAABIUZgAASFCYAQAgQWEGAIAEp2QkvP/++9Fs6NChzbgSAAB2F0+YAQAgQWEGAIAEhRkAABIUZgAASFCYAQAgQWEGAIAEhRkAABIUZgAASFCYAQAgQWEGAIAEhRkAABIUZgAASCioMGdZ1tTrgD1SOX/2y/m9U97K9bNfru8bCvnsF1SYa2trG70YaInK+bNfzu+d8laun/1yfd9QyGc/lxVQq+vr68OaNWtCRUVFyOVyJVkc7MmyLAu1tbWhsrIytGpVnt+5ZN9Tbsp939vzlJuG7PmCCjMAAJSr8vsjNAAANIDCDAAACQozAAAkKMwAAJCgMO/B6urqQt++fcMf/vCHguc89NBDYfDgwaG+vr4JVwY0peHDh4f77ruv4NevW7cu9OjRI7z++utNuCqgqdjzez6FuRkceuihIZfL7fRrwoQJyXl33XVX6N27dxg2bFgIIYTf//73ea+Ty+XCggULQgghjBw5MrRt2zbMnDmzyd8XELdt27Zw/fXXh969e4eOHTuGPn36hG9/+9u7PCB/zpw54Y033gijR4/ePlZTUxO+/OUvhwMPPDDss88+4eijjw6//OUvt+fdunULY8eODRMnTmyy9wOkPfnkk+GMM84IlZWVIZfLhdmzZxc0L9+enzZtWhgxYkTo3LlzyOVyYePGjTvMseebn8LcDBYsWBDWrl27/de8efNCCCF88YtfjM7JsixMmTIlXHjhhdvHhg0btsN11q5dGy666KLQu3fvMGTIkO2vO//888PkyZOb7g0BuzRp0qRw5513hilTpoT/+q//CpMmTQq33HJL+OEPf5icN3ny5DB+/PgdzgQdO3ZsePXVV8OcOXPCokWLwllnnRW+9KUvhRdeeGH7a8aPHx9mzpwZ1q9f32TvCYh79913w1FHHRWmTp3aoHn59vyWLVvCyJEjwze+8Y3oPHu+mWU0uyuuuCLr06dPVl9fH33NggULslatWmWbNm2Kvqauri7r3r17duONN+4wvnLlyiyEkC1durRkawYa5vTTT88uuOCCHcbOOuus7LzzzovOefPNN7NcLpctXrx4h/F99tkn+3//7//tMNa1a9fsRz/60Q5jvXv3zu65555GrhxorBBCNmvWrF2+Lrbn/+rxxx/PQgjZhg0b8ub2fPPxhLmZ1dXVhZ/85CfhggsuSP5LSvPnzw/9+vULFRUV0dfMmTMnvP3222H8+PE7jPfq1Sv07NkzzJ8/v2TrBhpm2LBh4dFHHw1//vOfQwghvPjii+Gpp54Ko0aNis556qmnQqdOncLhhx++07V+/vOfh/Xr14f6+vrws5/9LGzdujWMGDFih9cdd9xx9j20ILE9Xyh7vvm02d0LKDezZ88OGzduDOeff37ydStXrgyVlZXJ10yfPj2cdtpp4aCDDtopq6ysDCtXrmzMUoFGuPbaa8OmTZvCgAEDQuvWrcO2bdvCTTfdFM4777zonJUrV4aePXvu9E+03n///eGcc84JBxxwQGjTpk3o1KlTmDVrVujbt+8Or6usrNzh2zSAPVtszxfKnm8+CnMzmz59ehg1atQuy/B7770XOnToEM1ff/318PDDD4f7778/b96xY8ewZcuWRq0VKN79998fZs6cGe67775wxBFHhOrq6nDllVeGysrKMG7cuLxzYvv++uuvDxs3bgyPPPJI6NatW5g9e3b40pe+FObPnx8GDhy4/XX2PbQsu/pavyv2fPNRmJvRypUrwyOPPBIeeOCBXb62W7duYdGiRdF8xowZ4YADDgj/8A//kDdfv3596N69e9FrBRrn61//erj22mu3/+T7wIEDw8qVK8PNN98cLczdunULGzZs2GFs2bJlYcqUKWHx4sXhiCOOCCGEcNRRR4X58+eHqVOnhrvuumv7a+17aFny7fmGsOebj+9hbkYzZswIPXr0CKeffvouX1tVVRVeeeWVvEdQZVkWZsyYEcaOHRvatm27U75169awbNmyUFVVVZJ1Aw23ZcuWnf6atXXr1skz0quqqkJNTc0OX0D/+vSokGstXrzYvocWJN+ebwh7vvkozM2kvr4+zJgxI4wbNy60abPrB/snnnhi2Lx5c3j55Zd3yh577LGwfPnycNFFF+Wd++yzz4b27duHoUOHNnrdQHHOOOOMcNNNN4UHH3wwrFixIsyaNSt8//vfD2eeeWZ0TlVVVejWrVt4+umnt48NGDAg9O3bN1xyySXhT3/6U1i2bFm47bbbwrx588LnP//57a/bsmVLWLhwYTj11FOb8m0BEZs3bw7V1dWhuro6hBDC8uXLQ3V1dXjttdeic/Lt+RD+5+z16urqsHTp0hBCCIsWLQrV1dU7HCFnzzez3X1MR7l4+OGHsxBC9uqrrxY850tf+lJ27bXX7jQ+ZsyYbNiwYdF5F198cXbJJZcUtU6gNDZt2pRdccUVWa9evbIOHTpkhx12WPYv//Iv2fvvv5+cd80112SjR4/eYezPf/5zdtZZZ2U9evTIOnXqlA0aNGinY+buu+++rH///iV/H0Bh/noE3N//GjduXHJevj0/ceLEvNeaMWPG9tfY880rl2W7+Gen2G1eeumlcMopp4Rly5aFfffdt6A569atC/379w/PPfdc6N27dxOvECi1mpqacMQRR4Tnn38+HHLIIQXP++QnPxm+9rWvhXPPPbcJVweUmj3fMviWjD3YoEGDwqRJk8Ly5csLnrNixYpwxx13KMvQQh144IFh+vTpyb/G/Xvr1q0LZ511VhgzZkwTrgxoCvZ8y+AJMwAAJHjCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACW0KeVF9fX1Ys2ZNqKioCLlcrqnXBLtdlmWhtrY2VFZWhlatyvPPlfY95abc9709T7lpyJ4vqDCvWbMmHHzwwSVZHLQkq1atCgcddNDuXsZuYd9Trsp139vzlKtC9nxBf4SuqKgoyYKgpSnnz345v3fKW7l+9sv1fUMhn/2CCrO/mqFclfNnv5zfO+WtXD/75fq+oZDPfvl9kxYAADSAwgwAAAkKMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJLTZ3QvY3X74wx9Gs2OOOaaoaz700EN5x1euXBmdU1NTE80efvjhotYBAPD3BgwYEM2qq6uj2YIFC/KOf/rTn27skvZ4njADAECCwgwAAAkKMwAAJCjMAACQoDADAECCwgwAAAl71bFy7du3zzs+derU6JwLLrig5OsYOnRo3vEsy6Jz6uvro9lzzz0XzW644YZo9rvf/S6aAQDl6fjjj49mrVu3jmZHHnlk3vE+ffpE5yxbtqzwhe3BPGEGAIAEhRkAABIUZgAASFCYAQAgQWEGAIAEhRkAABL2qmPlrrnmmrzjTXF0XErq+LiYVq3if3Y57rjjolnqyLwxY8bkHU8dUwfsGYYPHx7NJk+eHM369++fd/yqq66KzrnzzjsLXxjQIowaNSqapY6kbdMmXg23bNmSd3zr1q2FL6yF8oQZAAASFGYAAEhQmAEAIEFhBgCABIUZAAAS9qpTMiorKxs854EHHohmL774YjTbvHlzNPuP//iPvOPt27ePzpk5c2Y0GzZsWDTr06dPNJs2bVre8WOPPTY6Z9u2bdEMysG+++4bzT788MNoFtuLRx55ZHROam+nTskYOHBgNIsZOnRoNHNKBrRcrVu3zjt+2WWXReccfPDB0SzVAx599NG846tXr47O2Vt4wgwAAAkKMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJOxVx8rFjkZ67bXXonNuueWWaNacR6yNGDEimj300EPR7NRTT41mgwcPzjv+1a9+NTpn6tSp0Qz2RJ06dco7Pnfu3KKuV1dXF8369u0bzXr27Jl3vEOHDtE5uVwummVZFs2KUVtbW9LrAXuGG2+8Me/45z73uaKut2DBgmg2duzYoq65N/CEGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABI2KuOlXvppZcaNN5S/Ou//ms0Sx1H165du7zj119/fXTOr3/962iWOp4PdpeOHTvmHf/0pz8dndMUx7lt3bo17/jmzZujc2bMmBHNDjjggGh2zjnnRLPWrVvnHU8dlwfs2QYMGBDNrrzyygZfL3VsbuyYupZgyJAh0ey5555r1LU9YQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEvaqY+X2Vk899VQ0u/XWW6PZv/zLv+Qd79GjR3TOoYceGs0cK8eeqLa2Nu/46aef3qzrWLFiRd7xTZs2ReesWbOmqHsdd9xx0axv374NXgew+3Xq1CmaTZw4sah5MT/96U+j2W9/+9sGX29PsWXLlia7tifMAACQoDADAECCwgwAAAkKMwAAJCjMAACQoDADAECCY+VauF/96lfRLHasXMrAgQOj2ZNPPtng60FTq6uryzv+0EMPNfNKSqtLly7RLHWMVC6XyzseO/YO2DOcccYZ0Wz06NENvt769euj2d13393g67UES5YsabJre8IMAAAJCjMAACQozAAAkKAwAwBAgsIMAAAJTslgB6mf0r3rrrui2bZt25piOVC2+vfvH80qKyujWZZlecdPPPHE6JwZM2YUvjCgaCNGjIhmP/7xj4u6ZmzPX3XVVdE5Tz31VFH3KmeeMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACY6Va+HeeuutaLZu3bq84926dYvO6du3bzRr165dNHvvvfeiGdBwAwcOLOn1Fi1aVNLrAQ13ww03RLP27dsXdc0pU6bkHS/2mDry84QZAAASFGYAAEhQmAEAIEFhBgCABIUZAAASFGYAAEgoi2PlunTpEs0qKyuLuuaHH34Yzf785z8Xdc1idO/ePZqljo+Luf3226OZo+Og+ZT6WLnm/H0Jytmll14azY4//viirrly5cpo9n//7/8t6po0jCfMAACQoDADAECCwgwAAAkKMwAAJCjMAACQoDADAEDCXnWs3KhRo/KOp45K69evX1H3qquri2bf+ta38o7PnTs3OufFF18sah3/63/9r6LmxSxatKik14O9SWq/xY6BW758eXTOeeedF80GDBhQ+MIKMGXKlGh2zDHHRLMbbrihpOuAvUXPnj3zjv+f//N/onPatm0bzVLH1d56663RbNOmTdGM0vGEGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABI2KuOlfvVr36Vd7xNm9K/zXbt2kWzm266Ke/4xIkTo3N+/etfR7MHH3wwml1zzTXRLOaDDz6IZu+//36Drwd7k3vuuSeanXPOOdFsn332afC9crlcNMuyrMHXCyF+5GXq9ywgv1R/+PGPf5x3/JBDDinqXqljXadOnVrUNSkdT5gBACBBYQYAgASFGQAAEhRmAABIUJgBACBhrzolY/Xq1XnHi/2J1bVr10az1E+znnrqqXnHUz+l/oUvfKGorBhLly6NZn/84x9Leq+mcPTRR0ezgw8+OO947AQV+Hv/+q//Gs0++tGPRrM+ffrkHV+3bl10TuqUjF69ekWzAw88MJo99thjecdTJ3zU1tZGMyhnRx55ZDQ77bTTGny9Dz/8MJp9+9vfbvD1aD6eMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACXvVsXI33nhj3vG77747OqdNm/h/goULF0aziy++OJp16NAh7/j8+fOjc1LHVZXaxz72sWgWO5ovhBCWLFkSzT7+8Y83ak0N0aVLl2gWO6arU6dOTbQa9jYrVqyIZqNGjYpmFRUVeceLPbItdjxcCOlj5QYMGFDSdUA5u/7660t6vX/7t3+LZrNmzSrpvSgtT5gBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgIS96li5GTNm5B1PHRP1ox/9KJp97nOfi2Zr1qyJZs8880ze8a5du0bnNKfUUXof+chHispK7bXXXotmDzzwQDS77bbbmmI5sEvFHNt26KGHRrNjjz22qHW0bdu2qHlQroYMGRLNUkdJFmP27NklvR7NxxNmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBhrzpWLubxxx+PZldddVU0u/XWW6NZ6jiooUOHFrSuv1VXVxfNXnjhhWh20003RbNXXnmlwetIueCCC6JZu3bt8o4vXLgwOmfBggXRbOPGjdFs3bp10QxaksMPPzyaderUqahr/vKXvyx2OVCWrr766mjWsWPHBl/vkUceiWZ//OMfG3w99gyeMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQEJZnJKRMmfOnKKywYMHR7NBgwY1eB1PPvlkNFuxYkWDr9cUvvGNb+zuJcBeJXXaTi6XK+qaa9euLXI1sPfq0aNHNCvmZKuU7373u9Hsgw8+KOm9aD6eMAMAQILCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACWV/rFyxqquri8oA/qpbt27RLMuyoq75+OOPF7sc2Gvtv//+0axXr14lvVd9fX1Jr8eewRNmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACDBsXIAu0m/fv2KmrdixYpo9tJLLxW5Gth7LV++PJrdcccd0eyyyy6LZuvXr887vmrVqsIXRovhCTMAACQozAAAkKAwAwBAgsIMAAAJCjMAACQozAAAkOBYOYAW5t13341mW7dubcaVQMtQV1cXzSZMmFBURnnxhBkAABIUZgAASFCYAQAgQWEGAIAEhRkAABIUZgAASHCsHEAL88tf/nJ3LwGgrHjCDAAACQozAAAkKMwAAJCgMAMAQILCDAAACbksy7JdvWjTpk1hv/32a471wB7lnXfeCZ07d97dy9gt7HvKVbnue3ueclXInveEGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgASFGQAAEhRmAABIUJgBACBBYQYAgISCCnOWZU29DtgjlfNnv5zfO+WtXD/75fq+oZDPfkGFuba2ttGLgZaonD/75fzeKW/l+tkv1/cNhXz2c1kBtbq+vj6sWbMmVFRUhFwuV5LFwZ4sy7JQW1sbKisrQ6tW5fmdS/Y95abc9709T7lpyJ4vqDADAEC5Kr8/QgMAQAMozAAAkKAwAwBAgsIMAAAJCvMerK6uLvTt2zf84Q9/KHjOQw89FAYPHhzq6+ubcGVAU7HvofwMHz483HfffQW/ft26daFHjx7h9ddfb8JV8bcU5mZw8803h2OPPTZUVFSEHj16hM9//vPh1Vdf3eW8u+66K/Tu3TsMGzYshBDCihUrwoUXXhh69+4dOnbsGPr06RMmTpwY6urqts8ZOXJkaNu2bZg5c2aTvR9g1w499NCQy+V2+jVhwoTkvL/f97///e/zXieXy4UFCxaEEOx72BM8+eST4YwzzgiVlZUhl8uF2bNnFzRvzpw54Y033gijR4/ePjZt2rQwYsSI0Llz55DL5cLGjRt3mNOtW7cwduzYMHHixBK+A1IU5mbwxBNPhAkTJoRnn302zJs3L3zwwQfh1FNPDe+++250TpZlYcqUKeHCCy/cPvbKK6+E+vr6cPfdd4eXX3453H777eGuu+4K3/jGN3aYe/7554fJkyc32fsBdm3BggVh7dq123/NmzcvhBDCF7/4xeicfPt+2LBhO1xn7dq14aKLLgq9e/cOQ4YM2f46+x52r3fffTccddRRYerUqQ2aN3ny5DB+/PgdzgHesmVLGDly5E5f3//W+PHjw8yZM8P69euLXjMNkNHs3nzzzSyEkD3xxBPR1yxYsCBr1apVtmnTpuS1brnllqx37947jK1cuTILIWRLly4tyXqBxrviiiuyPn36ZPX19dHXFLLv6+rqsu7du2c33njjDuP2Pew5QgjZrFmzdvm6N998M8vlctnixYvz5o8//ngWQsg2bNiQN+/du3d2zz33NGKlFMoT5t3gnXfeCSGE0LVr1+hr5s+fH/r16xcqKip2ea2/v06vXr1Cz549w/z58xu/WKDR6urqwk9+8pNwwQUXJP8FtUL2/Zw5c8Lbb78dxo8fv8O4fQ8tz1NPPRU6deoUDj/88KLmH3fccfZ8M1GYm1l9fX248sorw6c+9alw5JFHRl+3cuXKUFlZmbzW0qVLww9/+MNwySWX7JRVVlaGlStXNnq9QOPNnj07bNy4MZx//vnJ1xWy76dPnx5OO+20cNBBB+2U2ffQsqxcuTL07Nmz6H+K3Z5vPm129wLKzYQJE8LixYvDU089lXzde++9Fzp06BDNV69eHUaOHBm++MUvhq985Ss75R07dgxbtmxp9HqBxps+fXoYNWrULsvwrvb966+/Hh5++OFw//33583te2hZdrXnd8Webz6eMDejyy+/PPzmN78Jjz/+eN6nQ3+rW7duYcOGDXmzNWvWhBNPPDEMGzYsTJs2Le9r1q9fH7p3797oNQONs3LlyvDII4+Eiy66aJevTe37EEKYMWNGOOCAA8I//MM/5M3te2hZdrXnd8Webz4KczPIsixcfvnlYdasWeGxxx4LvXv33uWcqqqq8Morr4Qsy3YYX716dRgxYkQ45phjwowZM/L+Nc7WrVvDsmXLQlVVVcneA1CcGTNmhB49eoTTTz99l6+N7fsQ/uf3kRkzZoSxY8eGtm3b7pTb99DyVFVVhZqamqJL8+LFi+35ZqIwN4MJEyaEn/zkJ+G+++4LFRUVoaamJtTU1IT33nsvOufEE08MmzdvDi+//PL2sb+W5V69eoXvfe974a233tp+rb/17LPPhvbt24ehQ4c22XsCdq2+vj7MmDEjjBs3LrRps+vvgMu37//qscceC8uXL48+qbbvYffavHlzqK6uDtXV1SGEEJYvXx6qq6vDa6+9Fp1TVVUVunXrFp5++ukdxmtqakJ1dXVYunRpCCGERYsWherq6h2OkNuyZUtYuHBhOPXUU0v/ZtjZ7j2kozyEEPL+mjFjRnLel770pezaa6/d/r9nzJgRvdbfuvjii7NLLrmkKd4K0AAPP/xwFkLIXn311YLn/P2+/6sxY8Zkw4YNi86z72H3+usRcH//a9y4ccl511xzTTZ69OgdxiZOnLjL3nDfffdl/fv3b4J3Qj65LMvzd3/sEV566aVwyimnhGXLloV99923oDnr1q0L/fv3D88991xB3/oB7FnseygvNTU14YgjjgjPP/98OOSQQwqe98lPfjJ87WtfC+eee24Tro6/8i0Ze7BBgwaFSZMmheXLlxc8Z8WKFeGOO+7wRRNaKPseysuBBx4Ypk+fnvzWjb+3bt26cNZZZ4UxY8Y04cr4W54wAwBAgifMAACQoDADAECCwgwAAAkKMwAAJCjMAACQoDADAECCwgwAAAkKMwAAJCjMAACQ8P8BPjqlcAftZmAAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "ds, info = tfds.load('mnist', split='train', with_info=True)\n", "\n", "fig = tfds.show_examples(ds, info)" ] }, { "cell_type": "markdown", "metadata": { "id": "Y0iVVStvk0oI" }, "source": [ "## Access the dataset metadata\n", "\n", "All builders include a `tfds.core.DatasetInfo` object containing the dataset metadata.\n", "\n", "It can be accessed through:\n", "\n", "* The `tfds.load` API:\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:07.147922Z", "iopub.status.busy": "2024-12-14T12:42:07.147344Z", "iopub.status.idle": "2024-12-14T12:42:07.609154Z", "shell.execute_reply": "2024-12-14T12:42:07.608367Z" }, "id": "UgLgtcd1ljzt" }, "outputs": [], "source": [ "ds, info = tfds.load('mnist', with_info=True)" ] }, { "cell_type": "markdown", "metadata": { "id": "XodyqNXrlxTM" }, "source": [ "* The `tfds.core.DatasetBuilder` API:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:07.612570Z", "iopub.status.busy": "2024-12-14T12:42:07.612038Z", "iopub.status.idle": "2024-12-14T12:42:08.008414Z", "shell.execute_reply": "2024-12-14T12:42:08.007595Z" }, "id": "nmq97QkilxeL" }, "outputs": [], "source": [ "builder = tfds.builder('mnist')\n", "info = builder.info" ] }, { "cell_type": "markdown", "metadata": { "id": "zMGOk_ZsmPeu" }, "source": [ "The dataset info contains additional informations about the dataset (version, citation, homepage, description,...)." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:08.011773Z", "iopub.status.busy": "2024-12-14T12:42:08.011185Z", "iopub.status.idle": "2024-12-14T12:42:08.015475Z", "shell.execute_reply": "2024-12-14T12:42:08.014814Z" }, "id": "O-wLIKD-mZQT" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tfds.core.DatasetInfo(\n", " name='mnist',\n", " full_name='mnist/3.0.1',\n", " description=\"\"\"\n", " The MNIST database of handwritten digits.\n", " \"\"\",\n", " homepage='http://yann.lecun.com/exdb/mnist/',\n", " data_dir='gs://tensorflow-datasets/datasets/mnist/3.0.1',\n", " file_format=tfrecord,\n", " download_size=11.06 MiB,\n", " dataset_size=21.00 MiB,\n", " features=FeaturesDict({\n", " 'image': Image(shape=(28, 28, 1), dtype=uint8),\n", " 'label': ClassLabel(shape=(), dtype=int64, num_classes=10),\n", " }),\n", " supervised_keys=('image', 'label'),\n", " disable_shuffling=False,\n", " splits={\n", " 'test': ,\n", " 'train': ,\n", " },\n", " citation=\"\"\"@article{lecun2010mnist,\n", " title={MNIST handwritten digit database},\n", " author={LeCun, Yann and Cortes, Corinna and Burges, CJ},\n", " journal={ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist},\n", " volume={2},\n", " year={2010}\n", " }\"\"\",\n", ")\n" ] } ], "source": [ "print(info)" ] }, { "cell_type": "markdown", "metadata": { "id": "1zvAfRtwnAFk" }, "source": [ "### Features metadata (label names, image shape,...)\n", "\n", "Access the `tfds.features.FeatureDict`:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:08.018310Z", "iopub.status.busy": "2024-12-14T12:42:08.017780Z", "iopub.status.idle": "2024-12-14T12:42:08.022366Z", "shell.execute_reply": "2024-12-14T12:42:08.021732Z" }, "id": "RcyZXncqoFab" }, "outputs": [ { "data": { "text/plain": [ "FeaturesDict({\n", " 'image': Image(shape=(28, 28, 1), dtype=uint8),\n", " 'label': ClassLabel(shape=(), dtype=int64, num_classes=10),\n", "})" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "info.features" ] }, { "cell_type": "markdown", "metadata": { "id": "KAm9AV7loyw5" }, "source": [ "Number of classes, label names:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:08.025203Z", "iopub.status.busy": "2024-12-14T12:42:08.024690Z", "iopub.status.idle": "2024-12-14T12:42:08.029083Z", "shell.execute_reply": "2024-12-14T12:42:08.028430Z" }, "id": "HhfzBH6qowpz" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10\n", "['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']\n", "7\n", "7\n" ] } ], "source": [ "print(info.features[\"label\"].num_classes)\n", "print(info.features[\"label\"].names)\n", "print(info.features[\"label\"].int2str(7)) # Human readable version (8 -> 'cat')\n", "print(info.features[\"label\"].str2int('7'))" ] }, { "cell_type": "markdown", "metadata": { "id": "g5eWtk9ro_AK" }, "source": [ "Shapes, dtypes:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:08.031764Z", "iopub.status.busy": "2024-12-14T12:42:08.031338Z", "iopub.status.idle": "2024-12-14T12:42:08.039042Z", "shell.execute_reply": "2024-12-14T12:42:08.038359Z" }, "id": "SergV_wQowLY" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:absl:`FeatureConnector.dtype` is deprecated. Please change your code to use NumPy with the field `FeatureConnector.np_dtype` or use TensorFlow with the field `FeatureConnector.tf_dtype`.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:absl:`FeatureConnector.dtype` is deprecated. Please change your code to use NumPy with the field `FeatureConnector.np_dtype` or use TensorFlow with the field `FeatureConnector.tf_dtype`.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "{'image': (28, 28, 1), 'label': ()}\n", "{'image': tf.uint8, 'label': tf.int64}\n", "(28, 28, 1)\n", "\n" ] } ], "source": [ "print(info.features.shape)\n", "print(info.features.dtype)\n", "print(info.features['image'].shape)\n", "print(info.features['image'].dtype)" ] }, { "cell_type": "markdown", "metadata": { "id": "thMOZ4IKm55N" }, "source": [ "### Split metadata (e.g. split names, number of examples,...)\n", "\n", "Access the `tfds.core.SplitDict`:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:08.041835Z", "iopub.status.busy": "2024-12-14T12:42:08.041327Z", "iopub.status.idle": "2024-12-14T12:42:08.044950Z", "shell.execute_reply": "2024-12-14T12:42:08.044294Z" }, "id": "FBbfwA8Sp4ax" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'test': , 'train': }\n" ] } ], "source": [ "print(info.splits)" ] }, { "cell_type": "markdown", "metadata": { "id": "EVw1UVYa2HgN" }, "source": [ "Available splits:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:08.047658Z", "iopub.status.busy": "2024-12-14T12:42:08.047126Z", "iopub.status.idle": "2024-12-14T12:42:08.050901Z", "shell.execute_reply": "2024-12-14T12:42:08.050227Z" }, "id": "fRBieOOquDzX" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['test', 'train']\n" ] } ], "source": [ "print(list(info.splits.keys()))" ] }, { "cell_type": "markdown", "metadata": { "id": "iHW0VfA0t3dO" }, "source": [ "Get info on individual split:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:08.053692Z", "iopub.status.busy": "2024-12-14T12:42:08.053151Z", "iopub.status.idle": "2024-12-14T12:42:08.057249Z", "shell.execute_reply": "2024-12-14T12:42:08.056605Z" }, "id": "-h_OSpRsqKpP" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "60000\n", "['mnist-train.tfrecord-00000-of-00001']\n", "1\n" ] } ], "source": [ "print(info.splits['train'].num_examples)\n", "print(info.splits['train'].filenames)\n", "print(info.splits['train'].num_shards)" ] }, { "cell_type": "markdown", "metadata": { "id": "fWhSkHFNuLwW" }, "source": [ "It also works with the subsplit API:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2024-12-14T12:42:08.060009Z", "iopub.status.busy": "2024-12-14T12:42:08.059502Z", "iopub.status.idle": "2024-12-14T12:42:08.063597Z", "shell.execute_reply": "2024-12-14T12:42:08.062947Z" }, "id": "HO5irBZ3uIzQ" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "36000\n", "[FileInstruction(filename='gs://tensorflow-datasets/datasets/mnist/3.0.1/mnist-train.tfrecord-00000-of-00001', skip=9000, take=36000, examples_in_shard=60000)]\n" ] } ], "source": [ "print(info.splits['train[15%:75%]'].num_examples)\n", "print(info.splits['train[15%:75%]'].file_instructions)" ] }, { "cell_type": "markdown", "metadata": { "id": "YZp2XJwQQrI0" }, "source": [ "## Troubleshooting\n", "\n", "### Manual download (if download fails)\n", "\n", "If download fails for some reason (e.g. offline,...). You can always manually download the data yourself and place it in the `manual_dir` (defaults to `~/tensorflow_datasets/downloads/manual/`.\n", "\n", "To find out which urls to download, look into:\n", "\n", " * For new datasets (implemented as folder): [`tensorflow_datasets/`](https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/)`//checksums.tsv`. For example: [`tensorflow_datasets/datasets/bool_q/checksums.tsv`](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/datasets/bool_q/checksums.tsv).\n", "\n", " You can find the dataset source location in [our catalog](https://www.tensorflow.org/datasets/catalog/overview).\n", " * For old datasets: [`tensorflow_datasets/url_checksums/.txt`](https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/url_checksums)\n", "\n", "### Fixing `NonMatchingChecksumError`\n", "\n", "TFDS ensure determinism by validating the checksums of downloaded urls.\n", "If `NonMatchingChecksumError` is raised, might indicate:\n", "\n", " * The website may be down (e.g. `503 status code`). Please check the url.\n", " * For Google Drive URLs, try again later as Drive sometimes rejects downloads when too many people access the same URL. See [bug](https://github.com/tensorflow/datasets/issues/1482)\n", " * The original datasets files may have been updated. In this case the TFDS dataset builder should be updated. Please open a new Github issue or PR:\n", " * Register the new checksums with `tfds build --register_checksums`\n", " * Eventually update the dataset generation code.\n", " * Update the dataset `VERSION`\n", " * Update the dataset `RELEASE_NOTES`: What caused the checksums to change ? Did some examples changed ?\n", " * Make sure the dataset can still be built.\n", " * Send us a PR\n", "\n", "Note: You can also inspect the downloaded file in `~/tensorflow_datasets/download/`.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "GmeeOokMODg2" }, "source": [ "## Citation\n", "\n", "If you're using `tensorflow-datasets` for a paper, please include the following citation, in addition to any citation specific to the used datasets (which can be found in the [dataset catalog](https://www.tensorflow.org/datasets/catalog/overview)).\n", "\n", "```\n", "@misc{TFDS,\n", " title = { {TensorFlow Datasets}, A collection of ready-to-use datasets},\n", " howpublished = {\\url{https://www.tensorflow.org/datasets}},\n", "}\n", "```" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "tensorflow/datasets", "private_outputs": true, "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.21" } }, "nbformat": 4, "nbformat_minor": 0 }