{"id":16493,"date":"2024-10-24T20:51:08","date_gmt":"2024-10-25T03:51:08","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=16493"},"modified":"2024-10-29T05:52:44","modified_gmt":"2024-10-29T12:52:44","slug":"prepare-datasets-fine-tuning-ml-models","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/pt\/prepare-datasets-fine-tuning-ml-models\/","title":{"rendered":"Prepara\u00e7\u00e3o de conjuntos de dados para ajuste fino de modelos de ML: Um guia abrangente"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">O ajuste fino dos modelos de aprendizado de m\u00e1quina come\u00e7a com conjuntos de dados bem preparados. Este guia o orientar\u00e1 sobre como criar esses conjuntos de dados, desde a coleta de dados at\u00e9 a cria\u00e7\u00e3o de arquivos de instru\u00e7\u00f5es. Ao final, voc\u00ea estar\u00e1 equipado com conhecimentos pr\u00e1ticos e ferramentas para preparar conjuntos de dados de alta qualidade para suas tarefas de ajuste fino. <\/span><\/p>\n<p style=\"text-align: left;\"><span style=\"font-weight: 400;\">Esta postagem continua os guias de detalhes de <a href=\"https:\/\/www.couchbase.com\/blog\/pt\/guide-to-data-prep-for-rag\/\">Prepara\u00e7\u00e3o de dados para RAG<\/a>e <a href=\"https:\/\/www.couchbase.com\/blog\/pt\/rag-applications-with-vector-search-and-couchbase\/\">Cria\u00e7\u00e3o de aplicativos RAG de ponta a ponta<\/a> com a pesquisa vetorial do Couchbase.<\/span><\/p>\n<div id=\"attachment_16494\" style=\"width: 910px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-16494\" class=\"wp-image-16494 size-large\" src=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/image1-8-1024x256.png\" alt=\"\" width=\"900\" height=\"225\" srcset=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/image1-8-1024x256.png 1024w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/image1-8-300x75.png 300w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/image1-8-768x192.png 768w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/image1-8-1536x383.png 1536w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/image1-8-1320x330.png 1320w, https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/image1-8.png 1999w\" sizes=\"auto, (max-width: 900px) 100vw, 900px\" \/><p id=\"caption-attachment-16494\" class=\"wp-caption-text\">Vis\u00e3o geral de alto n\u00edvel<\/p><\/div>\n<h2>Coleta de dados<\/h2>\n<p><span style=\"font-weight: 400;\">A primeira etapa \u00e9 a coleta de dados de v\u00e1rias fontes. Isso envolve a coleta de informa\u00e7\u00f5es brutas que, posteriormente, ser\u00e3o limpas e organizadas em conjuntos de dados estruturados.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><span style=\"font-weight: 400;\">Para obter um guia detalhado e passo a passo sobre a prepara\u00e7\u00e3o de dados para a gera\u00e7\u00e3o aumentada de recupera\u00e7\u00e3o, consulte nossa publica\u00e7\u00e3o abrangente no blog:\u00a0<\/span><span style=\"font-weight: 400;\">\"Guia passo a passo para preparar dados para a gera\u00e7\u00e3o aumentada de recupera\u00e7\u00e3o\"<\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<h3>Nossa abordagem \u00e0 coleta de dados<\/h3>\n<p><span style=\"font-weight: 400;\">Em nossa abordagem, utilizamos v\u00e1rios m\u00e9todos para coletar todos os dados relevantes:<\/span><\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Raspagem da Web usando o Scrapy:<\/b>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><a href=\"https:\/\/docs.scrapy.org\/en\/latest\/\"><span style=\"font-weight: 400;\">Sucata<\/span><\/a><span style=\"font-weight: 400;\"> \u00e9 uma poderosa estrutura Python para extrair dados de sites. Ele permite que voc\u00ea escreva spiders que rastreiam sites e extraem dados com efici\u00eancia.\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Extra\u00e7\u00e3o de documentos do Confluence:<\/b>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Fizemos o download direto dos documentos armazenados em nosso espa\u00e7o de trabalho do Confluence. Mas isso tamb\u00e9m pode ser feito utilizando a fun\u00e7\u00e3o <\/span><a href=\"https:\/\/docs.atlassian.com\/atlassian-confluence\/REST\/6.6.0\/\"><span style=\"font-weight: 400;\">API do Confluence<\/span><\/a><span style=\"font-weight: 400;\"> que envolveria a cria\u00e7\u00e3o de scripts para automatizar o processo de extra\u00e7\u00e3o.<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Recupera\u00e7\u00e3o de arquivos relevantes dos reposit\u00f3rios Git:<\/b>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"2\"><span style=\"font-weight: 400;\">Foram criados scripts personalizados para clonar reposit\u00f3rios e extrair arquivos relevantes. Isso garantiu a coleta de todos os dados necess\u00e1rios armazenados em nossos sistemas de controle de vers\u00e3o.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">Ao combinar esses m\u00e9todos, garantimos um processo de coleta de dados abrangente e eficiente, cobrindo todas as fontes necess\u00e1rias.<\/span><\/p>\n<h2>Extra\u00e7\u00e3o de conte\u00fado de texto<\/h2>\n<p><span style=\"font-weight: 400;\">Depois que os dados s\u00e3o coletados, a pr\u00f3xima etapa crucial \u00e9 a extra\u00e7\u00e3o de texto de documentos, como p\u00e1ginas da Web e PDFs. Esse processo envolve a an\u00e1lise desses documentos para obter dados de texto limpos e estruturados.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Para obter etapas detalhadas e exemplos de c\u00f3digo sobre a extra\u00e7\u00e3o de texto dessas fontes, consulte nosso guia abrangente na postagem do blog:\u00a0<\/span><span style=\"font-weight: 400;\">\"Guia passo a passo para preparar dados para a gera\u00e7\u00e3o aumentada de recupera\u00e7\u00e3o\"<\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<h3>Bibliotecas usadas para extra\u00e7\u00e3o de texto<\/h3>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>HTML:<\/b><span style=\"font-weight: 400;\">\u00a0<\/span><a href=\"https:\/\/beautiful-soup-4.readthedocs.io\/en\/latest\/\"><span style=\"font-weight: 400;\">BeautifulSoup<\/span><\/a><span style=\"font-weight: 400;\"> \u00e9 usado para navegar em estruturas HTML e extrair conte\u00fado de texto.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>PDFs:<\/b><span style=\"font-weight: 400;\">\u00a0<\/span><a href=\"https:\/\/pypdf2.readthedocs.io\/en\/3.x\/\"><span style=\"font-weight: 400;\">PyPDF2<\/span><\/a><span style=\"font-weight: 400;\"> facilita a leitura de arquivos PDF e a extra\u00e7\u00e3o de texto de cada p\u00e1gina.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Essas ferramentas nos permitem transformar documentos n\u00e3o estruturados em dados de texto organizados, prontos para processamento posterior.<\/span><\/p>\n<h2>Cria\u00e7\u00e3o de dados JSON de amostra<\/h2>\n<p><span style=\"font-weight: 400;\">Esta se\u00e7\u00e3o se concentra na gera\u00e7\u00e3o de instru\u00e7\u00f5es para a cria\u00e7\u00e3o de conjuntos de dados usando fun\u00e7\u00f5es como <code>generate_content()<\/code>\u00a0e\u00a0<code>generate_instructions()<\/code>que derivam perguntas com base no conhecimento do dom\u00ednio.<\/span><\/p>\n<h3>Gera\u00e7\u00e3o de instru\u00e7\u00f5es (perguntas)<\/h3>\n<p><span style=\"font-weight: 400;\">Para gerar perguntas de instru\u00e7\u00e3o, seguiremos estas etapas:<\/span><\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Se\u00e7\u00f5es de peda\u00e7os:<\/b><span style=\"font-weight: 400;\">\u00a0O texto \u00e9 dividido semanticamente para garantir perguntas significativas e contextualmente relevantes.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Formule perguntas:<\/b><span style=\"font-weight: 400;\">\u00a0Esses blocos s\u00e3o enviados a um modelo de linguagem (LLM), que gera perguntas com base no bloco recebido.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Criar formato JSON:<\/b><span style=\"font-weight: 400;\">\u00a0Por fim, estruturaremos as perguntas e as informa\u00e7\u00f5es associadas em um formato JSON para facilitar o acesso e a utiliza\u00e7\u00e3o.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<h3>Amostra\u00a0<i>instructions.json<\/i><\/h3>\n<p><span style=\"font-weight: 400;\">Aqui est\u00e1 um exemplo do que o\u00a0<code>instructions.json<\/code>\u00a0pode ter a apar\u00eancia de um arquivo depois de gerar e salvar as instru\u00e7\u00f5es:<\/span><\/p>\n<pre class=\"nums:false lang:default decode:true\">[\r\n    \"Qual \u00e9 a import\u00e2ncia do KV-Engine no contexto do Magma Storage Engine?\",\r\n    \"Qual \u00e9 a import\u00e2ncia da Arquitetura no contexto do Magma Storage Engine?\"\r\n]<\/pre>\n<h3>Implementa\u00e7\u00e3o<\/h3>\n<p><span style=\"font-weight: 400;\">Para implementar esse processo:<\/span><\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Carregar conhecimento do dom\u00ednio<\/b><span style=\"font-weight: 400;\">Recuperar informa\u00e7\u00f5es espec\u00edficas do dom\u00ednio de um arquivo designado<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Gerar instru\u00e7\u00f5es<\/b><span style=\"font-weight: 400;\">: utilizar fun\u00e7\u00f5es como <code>generate_content()<\/code> para decompor dados e formular perguntas usando <code>generate_instructions()<\/code><\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Salvar perguntas<\/b><span style=\"font-weight: 400;\">: uso <code>save_instructions()<\/code> para armazenar perguntas geradas em um arquivo JSON<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<h4><em>gerar_conte\u00fado<\/em> fun\u00e7\u00e3o<\/h4>\n<p><span style=\"font-weight: 400;\">O\u00a0<code>gerar_conte\u00fado<\/code> A fun\u00e7\u00e3o tokeniza o conhecimento do dom\u00ednio em frases e gera perguntas l\u00f3gicas com base nessas frases:<\/span><\/p>\n<pre class=\"nums:false lang:default decode:true\">def generate_content(domain_knowledge, context):\r\n    questions = []\r\n    # Tokenizar o conhecimento do dom\u00ednio em senten\u00e7as\r\n    senten\u00e7as = nltk.sent_tokenize(domain_knowledge)\r\n\r\n    # Gerar perguntas l\u00f3gicas com base em senten\u00e7as\r\n    for sentence in sentences:\r\n        question = generate_instructions(sentence, context)\r\n        questions.append(question)\r\n\r\n    retornar perguntas<\/pre>\n<h3><em>gerar_instru\u00e7\u00f5es<\/em> fun\u00e7\u00e3o<\/h3>\n<p><span style=\"font-weight: 400;\">Essa fun\u00e7\u00e3o demonstra como gerar perguntas de instru\u00e7\u00e3o usando uma API de modelo de idioma:<\/span><\/p>\n<pre class=\"nums:false lang:default decode:true\">def generate_instructions(domain, context):\r\n    prompt = \"Gere uma pergunta a partir do conhecimento de dom\u00ednio fornecido que possa ser respondida com o conhecimento de dom\u00ednio fornecido. N\u00e3o crie nem imprima nenhuma lista numerada, nenhuma sauda\u00e7\u00e3o, imprima diretamente a pergunta.\"\r\n    url = 'https:\/\/localhost:11434\/api\/generate'\r\n    data = {\"model\": model, \"stream\": False, \"prompt\": f\"[DOMAIN] {domain} [\/DOMAIN] [CONTEXT] {context} [\/CONTEXTO] {prompt}\"}\r\n    response = requests.post(url, json=data)\r\n    response.raise_for_status()\r\n\r\n    return response.json()['response'].strip()<\/pre>\n<h3>Carregando e salvando conhecimento de dom\u00ednio<\/h3>\n<p><span style=\"font-weight: 400;\">Usamos duas fun\u00e7\u00f5es adicionais:\u00a0<code>load_domain_knowledge()<\/code>\u00a0para carregar o conhecimento de dom\u00ednio de um arquivo e save_instructions() para salvar as instru\u00e7\u00f5es geradas em um arquivo JSON.<\/span><\/p>\n<h3><em>load_domain_knowledge<\/em> fun\u00e7\u00e3o<\/h3>\n<p><span style=\"font-weight: 400;\">Essa fun\u00e7\u00e3o carrega o conhecimento de dom\u00ednio de um arquivo especificado.<\/span><\/p>\n<pre class=\"nums:false lang:default decode:true\">def load_domain_knowledge(domain_file):\r\n    with open(domain_file, 'r') as file:\r\n        domain_knowledge = file.read()\r\n    return domain_knowledge<\/pre>\n<h3><em>save_instructions<\/em> fun\u00e7\u00e3o<\/h3>\n<p><span style=\"font-weight: 400;\">Essa fun\u00e7\u00e3o salva as instru\u00e7\u00f5es geradas em um arquivo JSON:<\/span><\/p>\n<pre class=\"nums:false lang:default decode:true\">def save_instructions(instructions, filename):\r\n    with open(filename, 'w') as file:\r\n        json.dump(instructions, file, indent=4)<\/pre>\n<hr \/>\n<h2>Exemplo de uso<\/h2>\n<p><span style=\"font-weight: 400;\">Veja a seguir um exemplo que demonstra como essas fun\u00e7\u00f5es funcionam juntas:<\/span><\/p>\n<pre class=\"nums:false lang:default decode:true\"># Exemplo de uso\r\ndomain_file = \"domain_knowledge.txt\"\r\ncontexto = \"contexto de amostra\"\r\ndomain_knowledge = load_domain_knowledge(domain_file)\r\ninstru\u00e7\u00f5es = generate_content(domain_knowledge, context)\r\nsave_instructions(instructions, \"instructions.json\")<\/pre>\n<p><span style=\"font-weight: 400;\">Esse fluxo de trabalho permite a cria\u00e7\u00e3o e o armazenamento eficientes de perguntas para a prepara\u00e7\u00e3o do conjunto de dados.<\/span><\/p>\n<h2>Gera\u00e7\u00e3o de conjuntos de dados (treinamento, teste, valida\u00e7\u00e3o)<\/h2>\n<p><span style=\"font-weight: 400;\">Esta se\u00e7\u00e3o o orienta na cria\u00e7\u00e3o de conjuntos de dados para ajustar v\u00e1rios modelos, como o Mistral 7B, usando o Llama2 da Ollama. Para garantir a precis\u00e3o, voc\u00ea precisar\u00e1 do conhecimento do dom\u00ednio armazenado em arquivos como <code>dom\u00ednio.txt<\/code>.<\/span><\/p>\n<h3>Fun\u00e7\u00f5es Python para cria\u00e7\u00e3o de conjuntos de dados<\/h3>\n<h4>query_ollama Fun\u00e7\u00e3o<\/h4>\n<p><span style=\"font-weight: 400;\">Essa fun\u00e7\u00e3o solicita respostas e perguntas de acompanhamento ao modelo Llama 2 da Ollama com base em prompts espec\u00edficos e no contexto do dom\u00ednio:<\/span><\/p>\n<pre class=\"nums:false lang:default decode:true\">def query_ollama(prompt, domain, context='', model='llama2'):\r\n  url = 'https:\/\/localhost:11434\/api\/generate'\r\n  data = {\"model\": model, \"stream\": False, \"prompt\": f\"[DOMAIN] {domain} [\/DOMAIN] [CONTEXT] {context} [\/CONTEXTO] {prompt}\"}\r\n  response = requests.post(url, json=data)\r\n  response.raise_for_status()\r\n\r\n  followup_data = {\"model\": model, \"stream\": False, \"prompt\": response.json()['response'].strip() + \"Qual \u00e9 uma prov\u00e1vel pergunta ou solicita\u00e7\u00e3o de acompanhamento? Retorna apenas o texto de uma pergunta ou solicita\u00e7\u00e3o.\"}\r\n\r\n  followup_response = requests.post(url, json=followup_data)\r\n  followup_response.raise_for_status()\r\n  return response.json()['response'].strip(), followup_response.json()['response'].replace(\"\\\"\", \"\").strip()<\/pre>\n<h4><em>create_validation_file<\/em> fun\u00e7\u00e3o<\/h4>\n<p><span style=\"font-weight: 400;\">Essa fun\u00e7\u00e3o divide os dados em conjuntos de treinamento, teste e valida\u00e7\u00e3o, salvando-os em arquivos separados para o treinamento do modelo:<\/span><\/p>\n<pre class=\"nums:false lang:default decode:true\">def create_validation_file(temp_file, train_file, valid_file, test_file):\r\nwith open(temp_file, 'r') as file:\r\n    lines = file.readlines()\r\n\r\ntrain_lines = lines[:int(len(lines) * 0.8)]\r\ntest_lines = lines[int(len(lines) * 0.8):int(len(lines) * 0.9)]\r\nvalid_lines = lines[int(len(lines) * 0.9):]\r\n\r\nwith open(train_file, 'a') as file:\r\n    file.writelines(train_lines)\r\n\r\nwith open(valid_file, 'a') as file:\r\n    file.writelines(valid_lines)\r\n\r\nwith open(test_file, 'a') as file:\r\n    file.writelines(test_lines)<\/pre>\n<h3>Gerenciamento da cria\u00e7\u00e3o de conjuntos de dados<\/h3>\n<h4><em>principal<\/em> fun\u00e7\u00e3o<\/h4>\n<p><span style=\"font-weight: 400;\">A fun\u00e7\u00e3o principal coordena a gera\u00e7\u00e3o do conjunto de dados, desde a consulta ao Llama 2 do Ollama at\u00e9 a formata\u00e7\u00e3o dos resultados em arquivos JSONL para treinamento do modelo:<\/span><\/p>\n<pre class=\"nums:false lang:default decode:true\">def main(temp_file, instructions_file, train_file, valid_file, test_file, domain_file, context=''):\r\n# Verificar se o arquivo de instru\u00e7\u00f5es existe\r\nif not Path(instructions_file).is_file():\r\n    sys.exit(f'{instructions_file} not found.')\r\n\r\n# Verificar se o arquivo de dom\u00ednio existe\r\nif not Path(domain_file).is_file():\r\n    sys.exit(f'{domain_file} not found.')\r\n\r\n# Carregar conhecimento do dom\u00ednio\r\ndomain = load_domain(domain_file)\r\n\r\n# Carregar instru\u00e7\u00f5es do arquivo\r\nwith open(instructions_file, 'r') as file:\r\n    instructions = json.load(file)\r\n\r\n# Processar cada instru\u00e7\u00e3o\r\nfor i, instruction in enumerate(instructions, start=1):\r\n    print(f \"Processing ({i}\/{len(instructions)}): {instruction}\")\r\n      \r\n    # Consultar o modelo llama2 do Ollama para obter a resposta do modelo e a pergunta de acompanhamento\r\n    answer, followup_question = query_ollama(instruction, domain, context)\r\n      \r\n    # Formatar o resultado no formato JSONL\r\n    result = json.dumps({\r\n        'text': f'<s>[INST] {instruction}[\/INST] {answer}<\/s>[INST]{followup_question}[\/INST]'\r\n    }) + \"\\n\"\r\n      \r\n    # Grave o resultado em um arquivo tempor\u00e1rio\r\n    with open(temp_file, 'a') as file:\r\n        file.write(result)\r\n\r\n# Criar arquivos de treinamento, teste e valida\u00e7\u00e3o\r\ncreate_validation_file(temp_file, train_file, valid_file, test_file)\r\nprint(\"Feito! Arquivos JSONL de treinamento, teste e valida\u00e7\u00e3o criados.\")<\/pre>\n<h3>Usando essas ferramentas<\/h3>\n<p><span style=\"font-weight: 400;\">Para come\u00e7ar a refinar modelos como o Mistral 7B com o Llama 2 da Ollama:<\/span><\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Preparar o conhecimento do dom\u00ednio:<\/b><span style=\"font-weight: 400;\"> armazenar detalhes espec\u00edficos do dom\u00ednio em <code>dom\u00ednio.txt<\/code><\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Gerar instru\u00e7\u00f5es:<\/b><span style=\"font-weight: 400;\"> criar um arquivo JSON, <code>instructions.json<\/code>com prompts para cria\u00e7\u00e3o de conjuntos de dados<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Execute a fun\u00e7\u00e3o principal:<\/b><span style=\"font-weight: 400;\"> executar <code>principal()<\/code> com caminhos de arquivos para criar conjuntos de dados para treinamento e valida\u00e7\u00e3o de modelos<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">Essas fun\u00e7\u00f5es Python permitem que voc\u00ea desenvolva conjuntos de dados que otimizam os modelos de aprendizado de m\u00e1quina, melhorando o desempenho e a precis\u00e3o de aplicativos avan\u00e7ados.<\/span><\/p>\n<h2>Conclus\u00e3o<\/h2>\n<p><span style=\"font-weight: 400;\">Isso \u00e9 tudo por hoje! Com essas etapas, voc\u00ea agora tem o conhecimento e as ferramentas para aprimorar seu processo de treinamento de modelos de aprendizado de m\u00e1quina. Agradecemos sua leitura e esperamos que voc\u00ea tenha achado este guia valioso. N\u00e3o deixe de explorar nossos outros blogs para obter mais insights. Fique atento \u00e0 pr\u00f3xima parte desta s\u00e9rie e confira outros <a href=\"https:\/\/www.couchbase.com\/blog\/pt\/category\/vector-search\/\">blogs relacionados \u00e0 pesquisa de vetores<\/a>. Boa modelagem e at\u00e9 a pr\u00f3xima!<\/span><\/p>\n<h2>Refer\u00eancias<\/h2>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><a href=\"https:\/\/docs.scrapy.org\/en\/latest\/\"><span style=\"font-weight: 400;\">Documentos do m\u00f3dulo Python Scrapy<\/span><\/a><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><a href=\"https:\/\/docs.atlassian.com\/atlassian-confluence\/REST\/6.6.0\/\"><span style=\"font-weight: 400;\">Documentos da API REST do Confluence<\/span><\/a><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><a href=\"https:\/\/beautiful-soup-4.readthedocs.io\/en\/latest\/\"><span style=\"font-weight: 400;\">Sopa bonita 4 docs<\/span><\/a><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><a href=\"https:\/\/pypdf2.readthedocs.io\/en\/3.x\/\"><span style=\"font-weight: 400;\">Documenta\u00e7\u00e3o do m\u00f3dulo PyPDF<\/span><\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>Contribuintes<\/h2>\n<p style=\"padding-left: 40px;\"><a href=\"https:\/\/in.linkedin.com\/in\/sanjivanipatra\"><span style=\"font-weight: 400;\">Sanjivani Patra<\/span><\/a> &#8211;\u00a0<a href=\"https:\/\/www.linkedin.com\/in\/nishanth-vm?utm_source=share&amp;utm_campaign=share_via&amp;utm_content=profile&amp;utm_medium=android_app\"><span style=\"font-weight: 400;\">Nishanth VM<\/span><\/a> &#8211; <a href=\"https:\/\/www.linkedin.com\/in\/ashokkumaralluri\/\"><span style=\"font-weight: 400;\">Ashok Kumar Alluri<\/span><\/a><\/p>\n<p>&nbsp;<\/p>","protected":false},"excerpt":{"rendered":"<p>O ajuste fino dos modelos de aprendizado de m\u00e1quina come\u00e7a com conjuntos de dados bem preparados. Este guia o orientar\u00e1 sobre como criar esses conjuntos de dados, desde a coleta de dados at\u00e9 a cria\u00e7\u00e3o de arquivos de instru\u00e7\u00f5es. Ao final, voc\u00ea estar\u00e1 equipado com conhecimentos pr\u00e1ticos e ferramentas para preparar [...]<\/p>","protected":false},"author":85513,"featured_media":16495,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1814,1815,9973,9139,9937],"tags":[9231,9923,2140,9924],"ppma_author":[10012],"class_list":["post-16493","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-application-design","category-best-practices-and-tutorials","category-generative-ai-genai","category-python","category-vector-search","tag-data-science","tag-embeddings","tag-machine-learning","tag-rag-retrieval-augmented-generation"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.7.1 (Yoast SEO v25.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide - The Couchbase Blog<\/title>\n<meta name=\"description\" content=\"Create high-quality datasets for fine-tuning models with this guide on data gathering, text extraction, and instruction file generation.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.couchbase.com\/blog\/pt\/prepare-datasets-fine-tuning-ml-models\/\" \/>\n<meta property=\"og:locale\" content=\"pt_BR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide\" \/>\n<meta property=\"og:description\" content=\"Create high-quality datasets for fine-tuning models with this guide on data gathering, text extraction, and instruction file generation.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.couchbase.com\/blog\/pt\/prepare-datasets-fine-tuning-ml-models\/\" \/>\n<meta property=\"og:site_name\" content=\"The Couchbase Blog\" \/>\n<meta property=\"article:published_time\" content=\"2024-10-25T03:51:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-10-29T12:52:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2400\" \/>\n\t<meta property=\"og:image:height\" content=\"1256\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Sanjivani Patra - Software Engineer\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sanjivani Patra - Software Engineer\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutos\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/\"},\"author\":{\"name\":\"Sanjivani Patra - Software Engineer\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/bef3d836c397a5a2cd59c80a43070250\"},\"headline\":\"Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide\",\"datePublished\":\"2024-10-25T03:51:08+00:00\",\"dateModified\":\"2024-10-29T12:52:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/\"},\"wordCount\":919,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png\",\"keywords\":[\"data science\",\"embeddings\",\"Machine Learning (ML)\",\"RAG retrieval-augmented generation\"],\"articleSection\":[\"Application Design\",\"Best Practices and Tutorials\",\"Generative AI (GenAI)\",\"Python\",\"Vector Search\"],\"inLanguage\":\"pt-BR\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/\",\"name\":\"Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide - The Couchbase Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png\",\"datePublished\":\"2024-10-25T03:51:08+00:00\",\"dateModified\":\"2024-10-29T12:52:44+00:00\",\"description\":\"Create high-quality datasets for fine-tuning models with this guide on data gathering, text extraction, and instruction file generation.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#breadcrumb\"},\"inLanguage\":\"pt-BR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"pt-BR\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#primaryimage\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png\",\"width\":2400,\"height\":1256},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.couchbase.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"name\":\"The Couchbase Blog\",\"description\":\"Couchbase, the NoSQL Database\",\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"pt-BR\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\",\"name\":\"The Couchbase Blog\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"pt-BR\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"width\":218,\"height\":34,\"caption\":\"The Couchbase Blog\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/bef3d836c397a5a2cd59c80a43070250\",\"name\":\"Sanjivani Patra - Software Engineer\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"pt-BR\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/e4335546d16f0f88af0dd52b94ab89d7\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/fffb8781af2f43cc312cfd67e2873c3a93111077161855cc2b605e3733cb712b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/fffb8781af2f43cc312cfd67e2873c3a93111077161855cc2b605e3733cb712b?s=96&d=mm&r=g\",\"caption\":\"Sanjivani Patra - Software Engineer\"},\"url\":\"https:\/\/www.couchbase.com\/blog\/pt\/author\/sanjivanipatra\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide - The Couchbase Blog","description":"Create high-quality datasets for fine-tuning models with this guide on data gathering, text extraction, and instruction file generation.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.couchbase.com\/blog\/pt\/prepare-datasets-fine-tuning-ml-models\/","og_locale":"pt_BR","og_type":"article","og_title":"Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide","og_description":"Create high-quality datasets for fine-tuning models with this guide on data gathering, text extraction, and instruction file generation.","og_url":"https:\/\/www.couchbase.com\/blog\/pt\/prepare-datasets-fine-tuning-ml-models\/","og_site_name":"The Couchbase Blog","article_published_time":"2024-10-25T03:51:08+00:00","article_modified_time":"2024-10-29T12:52:44+00:00","og_image":[{"width":2400,"height":1256,"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png","type":"image\/png"}],"author":"Sanjivani Patra - Software Engineer","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Sanjivani Patra - Software Engineer","Est. reading time":"5 minutos"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#article","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/"},"author":{"name":"Sanjivani Patra - Software Engineer","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/bef3d836c397a5a2cd59c80a43070250"},"headline":"Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide","datePublished":"2024-10-25T03:51:08+00:00","dateModified":"2024-10-29T12:52:44+00:00","mainEntityOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/"},"wordCount":919,"commentCount":0,"publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png","keywords":["data science","embeddings","Machine Learning (ML)","RAG retrieval-augmented generation"],"articleSection":["Application Design","Best Practices and Tutorials","Generative AI (GenAI)","Python","Vector Search"],"inLanguage":"pt-BR","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/","url":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/","name":"Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide - The Couchbase Blog","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#primaryimage"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png","datePublished":"2024-10-25T03:51:08+00:00","dateModified":"2024-10-29T12:52:44+00:00","description":"Create high-quality datasets for fine-tuning models with this guide on data gathering, text extraction, and instruction file generation.","breadcrumb":{"@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#breadcrumb"},"inLanguage":"pt-BR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/"]}]},{"@type":"ImageObject","inLanguage":"pt-BR","@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#primaryimage","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/10\/blog-data-prep-for-ml-models.png","width":2400,"height":1256},{"@type":"BreadcrumbList","@id":"https:\/\/www.couchbase.com\/blog\/prepare-datasets-fine-tuning-ml-models\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.couchbase.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Preparing Datasets for Fine-Tuning ML Models: A Comprehensive Guide"}]},{"@type":"WebSite","@id":"https:\/\/www.couchbase.com\/blog\/#website","url":"https:\/\/www.couchbase.com\/blog\/","name":"Blog do Couchbase","description":"Couchbase, o banco de dados NoSQL","publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"pt-BR"},{"@type":"Organization","@id":"https:\/\/www.couchbase.com\/blog\/#organization","name":"Blog do Couchbase","url":"https:\/\/www.couchbase.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"pt-BR","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","width":218,"height":34,"caption":"The Couchbase Blog"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/bef3d836c397a5a2cd59c80a43070250","name":"Sanjivani Patra - Engenheiro de software","image":{"@type":"ImageObject","inLanguage":"pt-BR","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/e4335546d16f0f88af0dd52b94ab89d7","url":"https:\/\/secure.gravatar.com\/avatar\/fffb8781af2f43cc312cfd67e2873c3a93111077161855cc2b605e3733cb712b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/fffb8781af2f43cc312cfd67e2873c3a93111077161855cc2b605e3733cb712b?s=96&d=mm&r=g","caption":"Sanjivani Patra - Software Engineer"},"url":"https:\/\/www.couchbase.com\/blog\/pt\/author\/sanjivanipatra\/"}]}},"authors":[{"term_id":10012,"user_id":85513,"is_guest":0,"slug":"sanjivanipatra","display_name":"Sanjivani Patra - Software Engineer","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/fffb8781af2f43cc312cfd67e2873c3a93111077161855cc2b605e3733cb712b?s=96&d=mm&r=g","first_name":"Sanjivani","last_name":"Patra - Software Engineer","user_url":"","author_category":"","description":""}],"_links":{"self":[{"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/posts\/16493","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/users\/85513"}],"replies":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/comments?post=16493"}],"version-history":[{"count":0,"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/posts\/16493\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/media\/16495"}],"wp:attachment":[{"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/media?parent=16493"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/categories?post=16493"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/tags?post=16493"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/pt\/wp-json\/wp\/v2\/ppma_author?post=16493"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}