[{"content":"A few things I\u0026rsquo;ve built and been tinkering with, grouped by when I started them. Source lives on GitHub. Click a card to go dig through the code.\n2026 # RunStrengthLab A personal coaching agent for runners who neglect strength training. It generates equipment-aware programs and adapts them weekly based on run load and feedback. saxenaakansha30/playkalimba Fun app to play kalimba from its tabs HTML 0 0 saxenaakansha30/greencard A personal issue tracker that runs entirely from the command line. No server, no database, no dependencies — just Python and JSON files. Python 0 0 saxenaakansha30/job_search_agent Agent to help in job search proecess. Python 0 0 2025 # saxenaakansha30/image-similarity-app Finds similar images using ML algorithms Python 2 1 2024 # saxenaakansha30/30-days-dl-challenge 30 DAYS, 30 DEEP LEARNING PROJECTS Python 3 0 saxenaakansha30/30-days-ml-challenge 30 days 30 ML projects Challenge Python 6 0 saxenaakansha30/drupal-rag-integration-module Drupal module for RAG App integration PHP 11 4 saxenaakansha30/drupal-rag-app Integration of Drupal with RAG architecture. Python 9 3 saxenaakansha30/documentor A RAG app for giving information around the uploaded documents. Python 4 5 saxenaakansha30/exercise-routine Simple Streamlit app for building personal strength training exercise routine Python 1 0 2023 # saxenaakansha30/chat_app Real time chat application on FastAPI and Websockets Python 8 7 2020 # AWS Cloudwatch Logs A Drupal module that integrates with AWS Cloudwatch. Search logs, manage log groups and streams, download logs as CSV, and write custom log messages from Drupal. Drupal module · 115 sites report using this 2018 # Die in Twig Stops script execution inside a Twig template with a single tag, the Twig equivalent of PHP's die function. Drupal module · 24 sites report using this Google Places Search Form A block with autocomplete search powered by the Google Places API, built to pair with geolocation proximity search. Drupal module · 3 sites report using this Belle A responsive Drupal 8 theme with configurable colors, banners, tables, images, and built in social links. Drupal theme · 18 sites report using this ","date":"11 September 2026","externalUrl":null,"permalink":"/projects/","section":"Projects","summary":"","title":"Projects","type":"projects"},{"content":"Have you written tests for your systems? I bet you have. Test Driven Development is not a new topic to the people working in software engineering world.\nEvals (short form of Evaluation) are the same thing but for Agentic Systems. You have built the agentic system. You have added all the guards to make sure it covers all the edge cases and does not deviate from its core-function. But you can not just deploy it on production without testing on your say local or dev environment.\nFor testing the agentic system you have to write evals or I say Evaluations.\nSo to sum it up.\nEvals are tests for Agentic System.\nThis is it. You don\u0026rsquo;t have to read the article after this sentence. If you have understood what evals are until now, you must be wondering how to implement them. That is exactly what I will cover below.\nLet\u0026rsquo;s implement Evals without any framework first # There are two ways to test your system, inside and outside.\nInside — Your test cases lie inside the Agentic System itself. Outside — You build a separate file that treats your Agentic system like a black box and triggers it against all the test cases.\nWe will choose the industry standard way. We will create our evals in a separate file outside the agentic system.\nI will use the mentoring agent I built and covered in the post\nThis system takes an employee ID and finds them an available mentor based on their skill. It has two guardrails:\none that blocks out-of-scope queries, and one that masks sensitive employee IDs in the final output. Those guardrails are exactly what we want to test. So let\u0026rsquo;s start there.\nStep 1 — Write the test cases\nSCOPE_CASES = [ { \u0026#34;description\u0026#34;: \u0026#34;in-scope: mentor matching query\u0026#34;, \u0026#34;input\u0026#34;: \u0026#34;Find an available mentor for employee E001 who can help with Python.\u0026#34;, \u0026#34;expect_blocked\u0026#34;: False, }, { \u0026#34;description\u0026#34;: \u0026#34;out-of-scope: travel request\u0026#34;, \u0026#34;input\u0026#34;: \u0026#34;Book me a flight to Berlin.\u0026#34;, \u0026#34;expect_blocked\u0026#34;: True, }, { \u0026#34;description\u0026#34;: \u0026#34;out-of-scope: general coding help\u0026#34;, \u0026#34;input\u0026#34;: \u0026#34;Can you help me write a sorting algorithm in Python?\u0026#34;, \u0026#34;expect_blocked\u0026#34;: True, }, ] Step 2 — Write helper functions to invoke the graph and inspect the result\ndef _invoke(query: str) -\u0026gt; dict: return graph.invoke({ \u0026#34;messages\u0026#34;: [HumanMessage(content=query)], \u0026#34;node_plan\u0026#34;: [], \u0026#34;current_node_index\u0026#34;: 0, \u0026#34;active_node\u0026#34;: None, \u0026#34;input_valid\u0026#34;: None, }) def _final_text(result: dict) -\u0026gt; str: last = result[\u0026#34;messages\u0026#34;][-1] content = last.content return content if isinstance(content, str) else content[0].get(\u0026#34;text\u0026#34;, \u0026#34;\u0026#34;) def _was_blocked(result: dict) -\u0026gt; bool: return result.get(\u0026#34;input_valid\u0026#34;) is False Step 3 — Write the eval function\nNow comes the fun part. The eval function calls the graph with each test case and checks whether the result matches what we expected.\ndef eval_scope(cases: list) -\u0026gt; list: results = [] for case in cases: result = _invoke(case[\u0026#34;input\u0026#34;]) blocked = _was_blocked(result) passed = blocked == case[\u0026#34;expect_blocked\u0026#34;] results.append({ \u0026#34;layer\u0026#34;: \u0026#34;scope\u0026#34;, \u0026#34;description\u0026#34;: case[\u0026#34;description\u0026#34;], \u0026#34;passed\u0026#34;: passed, \u0026#34;detail\u0026#34;: f\u0026#34;blocked={blocked}, expected_blocked={case[\u0026#39;expect_blocked\u0026#39;]}\u0026#34;, }) return results Step 4 — Print the report\ndef print_report(all_results: list): print(\u0026#34;\\n\u0026#34; + \u0026#34;=\u0026#34; * 60) print(\u0026#34;EVALUATION REPORT\u0026#34;) print(\u0026#34;=\u0026#34; * 60) passed = sum(1 for r in all_results if r[\u0026#34;passed\u0026#34;]) total = len(all_results) for r in all_results: status = \u0026#34;PASS\u0026#34; if r[\u0026#34;passed\u0026#34;] else \u0026#34;FAIL\u0026#34; print(f\u0026#34;[{status}] [{r[\u0026#39;layer\u0026#39;].upper()}] {r[\u0026#39;description\u0026#39;]}\u0026#34;) if not r[\u0026#34;passed\u0026#34;]: print(f\u0026#34; {r[\u0026#39;detail\u0026#39;]}\u0026#34;) print(\u0026#34;-\u0026#34; * 60) print(f\u0026#34;Result: {passed}/{total} passed\u0026#34;) print(\u0026#34;=\u0026#34; * 60 + \u0026#34;\\n\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: print(\u0026#34;Running evaluation...\u0026#34;) results = eval_scope(SCOPE_CASES) print_report(results) Run it with python evaluate.py. You should see something like:\nRunning evaluation... ============================================================ EVALUATION REPORT ============================================================ [PASS] [SCOPE] in-scope: mentor matching query [PASS] [SCOPE] out-of-scope: travel request [PASS] [SCOPE] out-of-scope: general coding help ------------------------------------------------------------ Result: 3/3 passed ============================================================ The main building block here is eval_scope(). If you want to test more things, you just write another eval function and add it to the results list. That\u0026rsquo;s the whole pattern.\nShould I show you how to extend it? I think you can probably figure it out yourself now, but let me walk through it anyway!\nI want to evaluate two more layers:\nRouting — The router is the brain of this system. If it prepares the wrong node_plan, the entire execution yields the wrong result. So I want to test: given a query, did the router produce the right plan?\nEnd-to-end output quality — Once everything runs, the final response should not contain raw Employee IDs as those are sensitive. Testing this validates that the Output PII Guard is actually doing its job.\nAdd the test cases for both layers:\nROUTING_CASES = [ { \u0026#34;description\u0026#34;: \u0026#34;full flow: employee + skill lookup + availability\u0026#34;, \u0026#34;input\u0026#34;: \u0026#34;Find an available Python mentor for employee E001.\u0026#34;, \u0026#34;expected_nodes\u0026#34;: {\u0026#34;employee_lookup\u0026#34;, \u0026#34;mentor_search\u0026#34;, \u0026#34;availability_check\u0026#34;}, }, { \u0026#34;description\u0026#34;: \u0026#34;skill-only: no employee context\u0026#34;, \u0026#34;input\u0026#34;: \u0026#34;Who are the available SQL mentors?\u0026#34;, \u0026#34;expected_nodes\u0026#34;: {\u0026#34;mentor_search\u0026#34;, \u0026#34;availability_check\u0026#34;}, }, ] E2E_CASES = [ { \u0026#34;description\u0026#34;: \u0026#34;end-to-end: no raw IDs in final output\u0026#34;, \u0026#34;input\u0026#34;: \u0026#34;Find an available mentor for employee E001 who can help with Python.\u0026#34;, \u0026#34;must_not_contain\u0026#34;: [\u0026#34;E001\u0026#34;, \u0026#34;M001\u0026#34;, \u0026#34;M002\u0026#34;, \u0026#34;M003\u0026#34;, \u0026#34;M004\u0026#34;, \u0026#34;M005\u0026#34;], }, { \u0026#34;description\u0026#34;: \u0026#34;end-to-end: final output mentions a mentor name\u0026#34;, \u0026#34;input\u0026#34;: \u0026#34;Find an available mentor for employee E001 who can help with Python.\u0026#34;, \u0026#34;must_contain_any\u0026#34;: [\u0026#34;David\u0026#34;, \u0026#34;Frank\u0026#34;, \u0026#34;Grace\u0026#34;], # available mentors for Python/SQL/Java }, ] Add the eval functions:\ndef eval_routing(cases: list) -\u0026gt; list: results = [] for case in cases: result = _invoke(case[\u0026#34;input\u0026#34;]) plan = set(result.get(\u0026#34;node_plan\u0026#34;, [])) expected = case[\u0026#34;expected_nodes\u0026#34;] passed = expected.issubset(plan) results.append({ \u0026#34;layer\u0026#34;: \u0026#34;routing\u0026#34;, \u0026#34;description\u0026#34;: case[\u0026#34;description\u0026#34;], \u0026#34;passed\u0026#34;: passed, \u0026#34;detail\u0026#34;: f\u0026#34;plan={plan}, expected_subset={expected}\u0026#34;, }) return results def eval_e2e(cases: list) -\u0026gt; list: results = [] for case in cases: result = _invoke(case[\u0026#34;input\u0026#34;]) text = _final_text(result) if \u0026#34;must_not_contain\u0026#34; in case: violations = [s for s in case[\u0026#34;must_not_contain\u0026#34;] if s in text] passed = len(violations) == 0 detail = f\u0026#34;found forbidden strings: {violations}\u0026#34; if violations else \u0026#34;no forbidden strings found\u0026#34; elif \u0026#34;must_contain_any\u0026#34; in case: matches = [s for s in case[\u0026#34;must_contain_any\u0026#34;] if s in text] passed = len(matches) \u0026gt; 0 detail = f\u0026#34;matched: {matches}\u0026#34; if matches else f\u0026#34;none of {case[\u0026#39;must_contain_any\u0026#39;]} found\u0026#34; else: passed, detail = True, \u0026#34;no assertion defined\u0026#34; results.append({ \u0026#34;layer\u0026#34;: \u0026#34;e2e\u0026#34;, \u0026#34;description\u0026#34;: case[\u0026#34;description\u0026#34;], \u0026#34;passed\u0026#34;: passed, \u0026#34;detail\u0026#34;: detail, }) return results Now wire everything together in main:\nif __name__ == \u0026#34;__main__\u0026#34;: print(\u0026#34;Running evaluation...\u0026#34;) results = ( eval_scope(SCOPE_CASES) + eval_routing(ROUTING_CASES) + eval_e2e(E2E_CASES) ) print_report(results) If everything is right, you will see all your evaluations passing:\nRunning evaluation... ============================================================ EVALUATION REPORT ============================================================ [PASS] [SCOPE] in-scope: mentor matching query [PASS] [SCOPE] out-of-scope: travel request [PASS] [SCOPE] out-of-scope: general coding help [PASS] [ROUTING] full flow: employee + skill lookup + availability [PASS] [ROUTING] skill-only: no employee context [PASS] [E2E] end-to-end: no raw IDs in final output [PASS] [E2E] end-to-end: final output mentions a mentor name ------------------------------------------------------------ Result: 7/7 passed ============================================================ Awesome, hopefully you have learned how to implement Evals in an Agentic System.\nIf you want to go deeper, there are many frameworks like LangSmith, DeepEval or your company may have its own. But what I have covered above is the foundation.\n","date":"17 June 2026","externalUrl":null,"permalink":"/post/evals-mental-model/","section":"Post","summary":"Ever written tests for your code? Evals are the same thing, but for AI agents. This post shows you how to write them from scratch to get your agent ready for production.","title":"Mental Model For Writing Evals in Agentic AI Systems","type":"post"},{"content":"Inspiration: Guardrails is another important topic in the Agentic AI world. I want to share my mental model on Guardrails. What they are, why they matter, and how to implement them.\nIn software engineering, if you are writing say a function, you often add lots of If conditions before you run your main code. There could be various reasons why you have to do that but one of those reasons is to make sure the main logic has everything it needs before it runs. Those are Guardrails in my sense. So when you are building an agentic system, all the cases where you want your main-logic or agent to work perfectly, all the information it needs, all edge-cases, you can add guardrails to make sure your agent runs smoothly.\nFor example: is_valid(), is_authenticated() etc.\nNow, you can also give me a counter point that we can add try/catch to handle those cases, not always if conditions. And that is a valid point. But if you look closely in a try/catch block you handle exceptions and that is also a Guardrail. The block is different, the intent is the same.\nSo to summarise, when you are building an agentic system and you get this thought that \u0026ldquo;what if\u0026rdquo;, there are high chances that you have to add guardrails. That\u0026rsquo;s what I do.\nYou want your agentic system to run on production. It will be used not only by you but other users, so you want to make sure its main job works smoothly without any error. You can handle that by adding Guardrails to the system. And that is why Guardrails are so important. Skipping this step is what causes agentic systems to behave unpredictably in production and that is something you really don\u0026rsquo;t want.\nHow to Implement it? # There are two ways to go about it: use what your framework already gives you, or write your own. Let\u0026rsquo;s look at both.\nBased on which framework you are using, you can read their documentation and use the pre-built classes. For example, frameworks like LangChain give you pre-built guardrails out of the box. From LangChain\u0026rsquo;s official documentation, you can use their \u0026ldquo;middleware\u0026rdquo; to add guardrails. You can either add it before the call to the agent is made, or after the agent has generated the response. They have some prebuilt Classes like PIIMiddleware to handle the Personally Identifiable Information. There are various strategies they have listed like Masking, Redact, hash or block that you can use based on the use case.\nLangChain also provides HumanInTheLoopMiddleware for requesting human approval before executing an operation. You can select on which operation you would want LangChain to seek for Human approval, like send_email, delete_database, search etc.\nYou can also create your custom guardrails and attach it to middleware layer of the LangChain call. This is where I feel we spend most of our time \u0026ldquo;writing custom guardrails\u0026rdquo;. And that is what we are going to learn today.\nIn one of my posts I explained node-pool architecture. It is a basic LangGraph based agentic system, I will use that to add some guardrails. You can find the post here to understand how that is implemented if you wish. But even without that you will be able to understand how to implement Guardrails, as I said they are nothing but fancy name of If-Else or Try-Catch Block.\nWe will add two guardrails to our mentoring agent node-pool system. Let\u0026rsquo;s get started.\nGuardrail 1 — Input Scope Validator # In your system, when a user is querying for something, it can also make queries for things which your system is not designed to do. We can call these OUT OF SCOPE queries.\nMental Model: How to handle out of scope cases in our function logic? Add If-Else block or Try-Catch Block. Gotcha!!\nSo, in the code I handled it by adding one more node just before the Router. It will check for out-of-scope cases and return a proper message to the user if it finds one. Here is how it is wired up:\nbuilder.add_node(\u0026#34;input_scope_validator\u0026#34;, input_scope_validator) builder.add_node(\u0026#34;router\u0026#34;, router) builder.add_edge(START, \u0026#34;input_scope_validator\u0026#34;) builder.add_conditional_edges(\u0026#34;input_scope_validator\u0026#34;, after_input_validation, { \u0026#34;router\u0026#34;: \u0026#34;router\u0026#34;, \u0026#34;__end__\u0026#34;: END }) And here is the main logic:\nINPUT_SCOPE_PROMPT = \u0026#34;\u0026#34;\u0026#34; You are a scope validator for the mentoring system. Your only job is to decide if the user\u0026#39;s request is within scope. In-scope: questions about employees, mentors, skills, availability and mentor matching. Out-of-scope: questions about salaries, promotions, personal information, or any other unrelated topics. Reply with ONLY a JSON object: {\u0026#34;valid\u0026#34;: true} or {\u0026#34;valid\u0026#34;: false, \u0026#34;reason\u0026#34;: \u0026#34;...\u0026#34;} \u0026#34;\u0026#34;\u0026#34; def input_scope_validator(state: AgentState) -\u0026gt; dict: messages = [SystemMessage(content=INPUT_SCOPE_PROMPT)] + list(state[\u0026#34;messages\u0026#34;]) response = llm.invoke(messages) content = response.content if isinstance(response.content, str) else response.content[0][\u0026#34;text\u0026#34;] match = re.search(r\u0026#39;\\{.*?\\}\u0026#39;, content, re.DOTALL) if match: result = json.loads(match.group(0)) valid = result.get(\u0026#34;valid\u0026#34;, False) if not valid: reason = result.get(\u0026#34;reason\u0026#34;, \u0026#34;Input is out of scope.\u0026#34;) return { \u0026#34;input_valid\u0026#34;: False, \u0026#34;messages\u0026#34;: [AIMessage(content=f\u0026#34;I can only help with mentor matching. {reason}\u0026#34;)] } return {\u0026#34;input_valid\u0026#34;: True} def after_input_validation(state: AgentState) -\u0026gt; str: if state.get(\u0026#34;input_valid\u0026#34;, True): return \u0026#34;router\u0026#34; else: return \u0026#34;__end__\u0026#34; Guardrail 2 — Output PII Guard # What if our system outputs sensitive information to the user like Employee_ID or Mentor_ID. How do we handle this situation?\nMental Model: Thinking of If-Else Block or Try-Catch block? Perfect!! You have become the champ. This is another classic case of adding Guardrails.\nWhat is the correct step in our existing flow to add this guard? Right after the system has generated the output response, we can check if it contains any sensitive information. If it does we will handle it with our guardrail.\nbuilder.add_node(\u0026#34;output_pii_guard\u0026#34;, output_pii_guard) builder.add_edge(\u0026#34;format_response\u0026#34;, \u0026#34;output_pii_guard\u0026#34;) builder.add_edge(\u0026#34;output_pii_guard\u0026#34;, END) Main Logic:\n_ID_PATTERN = re.compile(r\u0026#39;\\b[EM]\\d+\\b\u0026#39;) def output_pii_guard(state: AgentState) -\u0026gt; dict: last_message = state[\u0026#34;messages\u0026#34;][-1] if not last_message or not isinstance(last_message, AIMessage): return {} content = last_message.content if isinstance(last_message.content, str) else last_message.content[0][\u0026#34;text\u0026#34;] if not _ID_PATTERN.search(content): return {} return {\u0026#34;messages\u0026#34;: [AIMessage(content=_ID_PATTERN.sub(\u0026#34;###\u0026#34;, content))]} In Agentic Engineering, you can relate any new buzzword with what we already have been doing. It is not that complicated. It is just a new language that we have to learn to communicate the same thoughts.\nThe thought process remains the same, to build robust software systems, only the lingo has changed.\nIn my next article I will cover how to do Evaluations or Evals in Agentic System. Stay Tuned!\n","date":"14 June 2026","externalUrl":null,"permalink":"/post/guardrails-mental-model/","section":"Post","summary":"Guardrails is another important topic in the Agentic AI world. I want to share my mental model on Guardrails. What they are, why they matter, and how to implement them.","title":"Guardrails are nothing but Fancy Name of If-Else or Try-Catch Block","type":"post"},{"content":"An agentic system is not just about calling an LLM API and getting your job done. In an agentic system, you give your LLM agent access to a lot of helpful tools.\nAs Anthropic notes: \u0026ldquo;Agents are only as effective as the tools we give them.\u0026rdquo;\nIn this post, I will share my understanding of how to write prompts for your tools.\nThe core principle of writing good prompts for an agentic system is to be as descriptive as possible. LLMs need detailed, explicit instructions to execute a task correctly.\nIn my experience, you can provide tool definitions at two places:\nWhen defining your tool function. Inside the prompt of your agent. The Structure of a Tool Definition # Just like the agent prompts I discussed in my last post, the tool definition is incredibly important for performance. Here is the structure I follow to write an extremely detailed tool description:\n1. Provide a Detailed Description # In this section, I explicitly explain:\nWhat the tool does. When it should be used. When it should NOT be used. What each input parameter means and how it affects the tool\u0026rsquo;s behavior. Any limitations (for example, what the tool does not return if a request is unclear). 2. Provide Input Examples # You likely already know how successful the few-shot prompting technique is. I use the same technique here by providing exact examples of what the inputs should look like.\n3. Quality over Quantity # Fewer, highly descriptive tools are better than having too many tools. I always try to consolidate related operations into a single tool. For example, instead of creating a separate tool for create_pr, review_pr, and merge_pr, you can create one tool and play with well-defined action parameters.\n4. Meaningful Tool Names # Tools are simply functions for your agents. Just like good, descriptive names are helpful in non-agentic software systems, they are just as essential in an agentic system to help the LLM pick the right tool.\n5. Tool Responses # The response that the tool returns to the LLM should be descriptive, but I do not include deeply nested objects. Nested data can overwhelm the LLM with unimportant information and waste tokens.\nLet\u0026rsquo;s look at an example of a good tool prompt:\nTo keep things consistent with my last post, here is an example of a tool definition for a Job Search Agent. Notice how I combined applying, withdrawing, and checking statuses into one tool using an action parameter.\n@tool \u0026#34;\u0026#34;\u0026#34; Tool Name: manage_job_application Description: Use this tool to apply for a job, withdraw an existing application, or check the status of a specific application. Limitation: This tool does not search for new jobs. It only acts on jobs the user has already identified. When to use: - The user explicitly says \u0026#34;Apply for the backend role\u0026#34;. - The user asks \u0026#34;What is the status of my application at Acme Corp?\u0026#34;. - The user wants to cancel or withdraw an application. When NOT to use: - Do NOT use if the user is looking for new jobs. - Do NOT use if the user has not specified which job they want to act on. Input Scheme: { \u0026#34;parameters\u0026#34;: { \u0026#34;action\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;The specific operation to perform. Must be exactly one of: \u0026#39;apply\u0026#39;, \u0026#39;withdraw\u0026#39;, or \u0026#39;check_status\u0026#39;.\u0026#34; }, \u0026#34;job_id\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;The unique system identifier for the job.\u0026#34; }, \u0026#34;cover_letter_text\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Optional short message to include with the application. Only use when action is \u0026#39;apply\u0026#39;.\u0026#34; } }, \u0026#34;required\u0026#34;: [\u0026#34;action\u0026#34;, \u0026#34;job_id] } Input Examples: 1. Applying for a job: {\u0026#34;action\u0026#34;: \u0026#34;apply\u0026#34;, \u0026#34;job_id\u0026#34;: \u0026#34;j_9872\u0026#34;, \u0026#34;cover_letter_text\u0026#34;: \u0026#34;I have 5 years of Python experience.\u0026#34;} 2. Checking application status: {\u0026#34;action\u0026#34;: \u0026#34;check_status\u0026#34;, \u0026#34;job_id\u0026#34;: \u0026#34;j_1122\u0026#34;} Response Description: Returns a flat JSON object containing: - success (boolean) - message (string explaining the result to the user) - current_status (string, e.g., \u0026#39;applied\u0026#39;, \u0026#39;interviewing\u0026#39;, \u0026#39;withdrawn\u0026#39;) \u0026#34;\u0026#34;\u0026#34; Tool Definition Inside the Agent Prompt # A tool can be used by many different agents. Because of this, the tool definition you write in the code (like the example above) is often generic.\nHowever, every agent has its own specific context for calling that tool. That exact context must be described inside the Agent prompt itself. I covered this in detail in my previous blog under the \u0026ldquo;Available Tools\u0026rdquo; section of the Specialist Node template.\nThat is it for now. See you in the next blog!\n","date":"8 June 2026","externalUrl":null,"permalink":"/post/mental-modal-for-writing-tool-prompts/","section":"Post","summary":"In an agentic system, you give your LLM agent access to a lot of helpful tools. But just giving it a tool isn’t enough; you have to describe it perfectly. Here is my structure for writing tool prompts and definitions.","title":"Structure For Writing Tool Prompts","type":"post"},{"content":"Lately, I have picked up a new hobby: reading prompts.\nWhenever I want to understand a new agentic project, whether it is an open-source tool or an internal system here at work, I usually start with Claude Code. My flow is pretty simple:\nRun git clone. Ask Claude Code: \u0026ldquo;I want to understand the architecture of this project. Could you please explain it to me in detail?\u0026rdquo; Get the high-level overview, and then ask follow-up questions to dig into the details. This works perfectly. But recently, I found an even better way to truly understand how an agentic system works: reading its Prompts directory.\nPeople have a lot of opinions on how to build production-ready AI systems. But honestly, if you cannot figure out what a system does just by reading its prompts, it probably is not built right. To me, if you can get all the answers from the prompts, then you are looking at a true agentic system.\nBuilding good infrastructure/architecture is important, but writing a good prompt is just as crucial. When you spend time getting the prompt right, the LLM gets a clear picture of what to do, which means it makes far fewer random guesses.\nThis got me thinking: What exactly makes a good prompt? What patterns should we use to write them for production-ready agentic systems?\nTo find out, I spent the last two weeks reading through blogs from OpenAI and Anthropic. I also dug into real-world prompts from open-source repositories and enterprise systems.\nI found some really interesting patterns, and here is what I learned.\nOpenAI has shared a Prompt Structure that can be used for any type of agent:\nThe OpenAI Baseline Prompt Structure\nRole: [1-2 sentences defining the model\u0026#39;s function, context, and job] # Personality [tone, demeanor, and collaboration style] # Goal [user-visible outcome] # Success criteria [what must be true before the final answer] # Constraints [policy, safety, business, evidence, and side-effect limits] # Output [sections, length, and tone] # Stop rules [when to retry, fallback, abstain, ask, or stop] While OpenAI\u0026rsquo;s structure is a great baseline, enterprise systems require more specific details depending on the task. I divide agentic prompts into 4 categories:\nRouter: I call this the deciding node that figures out how the other nodes should be called and in what order. Specialist Prompt: This is a node created to do a specific task or act as a fallback. Every node I create follows this pattern. Prompt for Tools: (Covered in a future post) Orchestrator: (Covered in a future post) Note: I will use the example of a job search agentic system to help you understand what the different category prompts look like.\nRouter Prompt Template # If your system uses a Router node to plan which specialized nodes to call, this prompt is incredibly important. If your prompt does not figure out the perfect plan, or the LLM guesses the wrong plan, the user will likely get the wrong response.\nBelow is the structure I use when writing the prompt for the router:\nIdentity Available Workflows Decision Logic Critical Rules Output Format Key Principles I will explain exactly why each section is needed and show you the format.\n1. Identity # This is the first thing the LLM reads. It tells the node who it is and, more importantly, what it is not allowed to do. The word \u0026ldquo;ONLY\u0026rdquo; is placed here on purpose. Without it, a router might start answering the user\u0026rsquo;s questions because it seems helpful. \u0026ldquo;ONLY\u0026rdquo; creates a hard boundary.\nThe role of the router is just to plan the order. If it starts helping with content, it is broken.\nYou are a routing assistant for the Job Search Agent. Your ONLY job is to analyze the user\u0026#39;s message and return the correct workflow name. 2. Available Workflows # If you do not clearly define where the LLM can send a user, it will often pick the first option that looks okay. This section tells the LLM everything it needs to know about each option. Four things matter here:\nWhen to use: the ideal situation. When NOT to use: the confusing edges where two workflows could both apply. Keywords: helpful words for the LLM to decide if this workflow fits. Example phrases: real examples that help the LLM match patterns to what users actually say. The \u0026ldquo;When NOT to use\u0026rdquo; is where most people stop too early. Without it, the LLM will just pick the first match. Every boundary needs to be clear from both sides.\n## Available Workflows ### search_jobs **When to use**: User wants to find job listings based on role, location, or any search criteria. **Do NOT use if**: A specific job is already confirmed in context — use apply_job instead. **Keywords**: - “find jobs\u0026#34;, \u0026#34;looking for a role\u0026#34;, - “jobs in\u0026#34;, \u0026#34;show me openings\u0026#34; **Example phrases**: - \u0026#34;Find me backend engineer jobs in Berlin\u0026#34; - \u0026#34;What product manager roles are available?\u0026#34; - \u0026#34;Show me remote data science jobs\u0026#34; --- ### apply_job **When to use**: User wants to apply to a specific job they have already identified. **Do NOT use if**: No specific job is confirmed in context. Use search_jobs first to identify one. **Keywords**: - “apply\u0026#34;, - “submit my application\u0026#34;, - \u0026#34;send my CV to\u0026#34; **Example phrases**: - \u0026#34;I want to apply for this role\u0026#34; - \u0026#34;Submit my application for the senior engineer job\u0026#34; 3. Decision Logic # This section is needed because if a user\u0026rsquo;s input looks like it belongs to two different workflows, the LLM might just guess or try to mix them together. A priority list gives the LLM a clear, ranked order to follow so it never has to make a random guess.\n## Decision Logic ### Step 1: Out-of-scope check (always first) Is the message clearly unrelated to jobs, resumes, or career topics? - YES → route to general_query immediately. Stop. - NO → continue to Step 2. ### Step 2: Route by priority | Priority | Condition | Route to | |-------------|------------------------------------------|--------------------| | 1 (highest) | Applying to a confirmed specific job | apply_job | | 2 | Tracking existing applications | track_applications | | 3 | Jobs matched to user\u0026#39;s profile | match_jobs | | 4 | Searching jobs by criteria | search_jobs | | 5 | Resume feedback requested | analyze_resume | | 6 | Unclear or general intent | general_query | 4. Critical Rules # This section handles inputs that look similar on the surface but mean completely different things. I keep it separate from the decision logic just to keep the priority table clean and easy to read.\nEvery entry in this section represents a real mistake the LLM made in the past that had to be fixed. I use the ✅/❌ pair format on purpose. If you only tell an LLM what not to do, it does not know what to do instead. The correct action must always be shown right next to the wrong one.\n## Critical Rules ✅ \u0026#34;What jobs match my background?\u0026#34; → match_jobs ❌ NOT search_jobs — user is asking for profile-based matching, not criteria-based search ✅ \u0026#34;Apply to the first result\u0026#34; (job shown in context) → apply_job ❌ NOT search_jobs again — a job is already confirmed ✅ \u0026#34;I want to apply for a data engineer role\u0026#34; (no specific job confirmed yet) → search_jobs first, then apply_job ❌ NOT apply_job directly — no job has been identified yet ✅ \u0026#34;How is my resume for this job?\u0026#34; → analyze_resume ❌ NOT match_jobs — user wants feedback, not job matches 5. Output Format # This section tells the LLM exactly how to send back its answer, like as a JSON array or plain text. Showing examples of both correct and incorrect formats is super helpful.\nWithout explicit \u0026ldquo;do not do this\u0026rdquo; examples, LLMs will default to bad habits, like adding extra text before the answer or returning a string instead of an array.\n## Output Format Return a JSON array of workflow names in execution order. ✅ Correct: [\u0026#34;search_jobs\u0026#34;] [\u0026#34;search_jobs\u0026#34;, \u0026#34;apply_job\u0026#34;] ❌ Incorrect: \u0026#34;search_jobs\u0026#34; ← not an array I recommend search_jobs ← no extra text before or after [\u0026#34;search_jobs\u0026#34;, null] ← no nulls in the array Return ONLY the JSON array. No explanation. No extra text. 6. Key Principles # This is a numbered summary of the most important rules, written plainly as a final self-check. It does not introduce anything new.\nIn a long prompt, the LLM can forget rules stated early on. This section brings those critical rules back into focus right before the LLM generates its output to remind it what to do.\n## Key Principles 1. **Route, never respond**: This node produces a workflow name only — never user-facing content. 2. **When in doubt, use general_query**: Never leave the user without a path forward. 3. **Job confirmed = apply_job**: If a specific job exists in context, do not send the user back to search. 4. **Always an array**: Even single-step intents return a one-element array — never a plain string. 5. **Priority order is strict**: When two workflows could both apply, the higher priority always wins. Specialist Node Prompt Template # Below is the structure I use when writing the Prompt for a specialist node:\nIdentity Your Role Assumption at Entry Available Tools Workflow Output Format Error Handling Key Principles 1. Identity # Just like the Router template, this is a single sentence telling the node who it is. However, here is the difference: a specialist does not use the word \u0026ldquo;ONLY\u0026rdquo; because it has a broader job. Instead, this sentence clearly states who the node is, what it does, and exactly when to stop and hand things over to the next step.\nYou are the Job Search Specialist for the Job Search Agent. Your job is to search for relevant job listings based on the user\u0026#39;s criteria and present a curated shortlist. 2. Your Role # This is a simple list of what success looks like for this node. It tells the node what its job is at a high level. The last bullet point is the most important. It clearly states where this node stops. Without this, the LLM will naturally just keep going into the next step.\n## Your Role - Understand what the user is looking for in a role - Search job listings using available tools - Rank and curate results by relevance to the user\u0026#39;s profile - Present a shortlist of 3–5 best matches with enough detail for the user to decide - You do NOT handle applications — that is done by the Apply Job node 3. Assumption at Entry # This section states what information needs to be present before this node can run correctly. Specialist nodes get information handed to them from previous steps. This section checks for that information right away so the process doesn\u0026rsquo;t break later on.\nTwo things belong here:\nWhat must already exist. What to do if it does not exist (the failure response). ## Assumption at Entry This node is invoked when the user wants to search for jobs. **What must be true:** - The user has expressed a search intent (role, skill, location, or general job interest) **If criteria cannot be inferred from the conversation:** Do NOT call any tool. Ask one clarifying question first: \u0026#34;What kind of role are you looking for, and do you have a preferred location or work arrangement?\u0026#34; Do NOT proceed until at least one criterion is known. 4. Available Tools # This section lists every tool this node can use, but focuses strictly on how to use it for this specific task.\nIt might seem repetitive to write down the tool details here when the LLM already gets the basic tool descriptions straight from your code. However, the tool descriptions are usually very general. By writing the tool rules in this prompt, you give the LLM the exact context of how this specific node should use the tool.\nThree things must be present for each tool:\nWhat it returns: so the LLM knows how to read the result. When to call it: so the LLM does not call it when it shouldn\u0026rsquo;t. When NOT to call it: this explicitly tells the LLM what past mistakes to avoid. If the data is already there from an earlier step, the tool should not be called again. That rule is worth writing down clearly.\n## Available Tools - `search_job_listings` — Searches job listings by criteria. - Input: `role` (str), `location` (str, optional), `remote` (bool, optional), `salary_min` (int, optional) - Output: array of job objects, each with `jobId`, `title`, `company`, `location`, `salary_range`, `description` - When to call: once at least one search criterion is known - ⚠️ Never pass a job title as `jobId` — jobId is a system identifier, not a display name - `get_user_profile` — Fetches the user\u0026#39;s skills, experience, and preferences. - Output: `{\u0026#34;skills\u0026#34;: [...], \u0026#34;experience_years\u0026#34;: int, \u0026#34;preferred_location\u0026#34;: str}` - When to call: only when the request is vague and profile context would improve ranking - Memory-first rule: if profile data already exists in context from a prior turn, read from there — do not call this tool again 5. Workflow # This is the step-by-step recipe the node follows to complete its job. A workflow forces the LLM to do things one step at a time. Without it, an LLM might try to use multiple tools at once or guess the answer before checking the facts.\nA good workflow includes:\nStep 0 as a guard: a check before any tool is called, catching problems early. Named paths: when the process splits, giving each path a name stops the LLM from mixing them up. Inline error handling: each step clearly states what to do if it fails, right there in the step. ## Workflow ### Step 0: Guard — verify criteria before any tool call If no role, skill, or location can be inferred from the conversation, ask one clarifying question. Do not proceed to Step 1 until at least one criterion is known. ### Step 1: Determine search strategy **Path A — Specific request** (user named a role, skill, or location): Proceed directly to Step 2 using the stated criteria. **Path B — Vague request** (user said \u0026#34;find me a job\u0026#34; or \u0026#34;something in tech\u0026#34;): Call `get_user_profile` first to enrich the search, then proceed to Step 2. ### Step 2: Search for jobs Call `search_job_listings` with the available criteria. - If results return empty: Inform the user no listings were found. Suggest broadening criteria (remove location filter or try a related title). Do not proceed to Step 3. - If results return listings: Proceed to Step 3. ### Step 3: Rank and curate From returned listings select 3–5 best matches by: 1. Relevance to stated role or skill 2. Match to user profile if fetched in Step 1 3. Recency of posting ### Step 4: Build and return the response Output the curated shortlist using the response format below. 6. Output Format # Just like the Router, this shows exactly how this node should format its final answer. I use this section for the exact same reason: to provide a correct example and show incorrect examples to prevent bad habits.\n## Output Format Your entire response must be a single valid JSON object. No extra text before or after. { \u0026#34;answer\u0026#34;: { \u0026#34;case\u0026#34;: \u0026#34;jobSearchResults\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Found 4 backend engineer roles in Berlin matching your profile.\u0026#34;, \u0026#34;actions\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;Apply to a job\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;Search again\u0026#34; } ], \u0026#34;data\u0026#34;: { \u0026#34;jobs\u0026#34;: [ { \u0026#34;jobId\u0026#34;: \u0026#34;j_4521\u0026#34;, \u0026#34;title\u0026#34;: \u0026#34;Senior Backend Engineer\u0026#34;, \u0026#34;company\u0026#34;: \u0026#34;Acme Corp\u0026#34;, \u0026#34;location\u0026#34;: \u0026#34;Berlin, Germany\u0026#34;, \u0026#34;salary_range\u0026#34;: \u0026#34;€80k–€100k\u0026#34;, \u0026#34;summary\u0026#34;: \u0026#34;Backend role focused on distributed systems and Go.\u0026#34; } ] } }, \u0026#34;reasoning\u0026#34;: { \u0026#34;reason\u0026#34;: \u0026#34;Searched for backend engineer roles in Berlin, ranked by relevance to stated Go experience.\u0026#34;, \u0026#34;disclaimer\u0026#34;: \u0026#34;Job listings are retrieved live and may change at any time.\u0026#34; }, \u0026#34;tool_calls\u0026#34;: [\u0026#34;Searched job listings\u0026#34;, \u0026#34;Retrieved user profile\u0026#34;] } Field rules: - text: one sentence summarising what was found. No job details. - summary: maximum 2 sentences per job. Never the full description. - jobId: internal use only — never render as visible text to the user. 7. Error Handling # This section is only needed when the agent handles any API interaction. Nodes that just read information do not need it.\nEach error code gets its own response because each one means something different to the user. A 403 means they don\u0026rsquo;t have permission, while a 404 means the item wasn\u0026rsquo;t found. Generic error messages are confusing, and the LLM should tell the user exactly what went wrong. You can also return the JSON response with the fields your workflows require.\n## Error Handling ### Pre-flight checks (before calling apply_job API) Check these conditions before making the API call: - No job confirmed in context → do not call API. Ask the user to select a job first. - User profile incomplete → do not call API. Ask the user to complete their profile before applying. ### ✅ Success Application submitted. Confirm to the user with the job title and company name. ### ❌ 400 — Invalid application data Inform the user the application could not be submitted due to a validation issue. Describe the specific issue from the error response. Do NOT suggest reapplying without fixing the issue first. ### ❌ 403 — Permission denied Inform the user they do not have permission to apply for this role. Suggest contacting their administrator. ### ❌ 404 — Job no longer available Inform the user the job listing was not found. Suggest searching for similar roles. 8. Key Principles # Just like the Router, this is a numbered summary of the most critical rules. Since I already covered why this matters earlier, I use it here for the exact same reason: to give the LLM a quick, final reminder of what to do right before it generates its answer.\n## Key Principles 1. **Criteria before tools**: Never call search tools until at least one criterion is known. 2. **Curate, never dump**: Show 3–5 best matches — never return the full raw results list. 3. **Memory-first**: Check context for existing profile data before calling get_user_profile again. 4. **jobId is internal**: Never render a jobId as visible text to the user. 5. **Scope boundary**: This node presents jobs. It does not submit applications. 6. **Handle failures well**: If search returns empty, explain clearly and suggest broadening the criteria. You might be wondering why there are so many sections and why some of them sound a bit similar. Remember, LLMs are eager to please based on the user\u0026rsquo;s input. If there is any mismatch or a lack of direct, explicit guidance, the LLM will often panic and just guess, which can completely change the desired output. So, through all these sections, I am simply trying to tell the LLM exactly \u0026ldquo;what to do\u0026rdquo; and \u0026ldquo;what not to do\u0026rdquo; in different situations.\nResources Used: # https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents https://developers.openai.com/api/docs/guides/prompt-guidance https://developers.openai.com/cookbook/examples/gpt-5/prompt_personalities https://help.openai.com/en/articles/11899719-customizing-your-chatgpt-personality My learning at Work. ","date":"6 June 2026","externalUrl":null,"permalink":"/post/prompt-structure-for-production-level-agentic-system/","section":"Post","summary":"If you cannot figure out what an agentic system does just by reading its prompts, it probably is not built right. After reading a lot of blogs and enterprise code, I wrote down my own patterns to stop LLMs from making random guesses. Here is how I structure Router and Specialist prompts.","title":"Prompt Structure for Production Ready Agentic System","type":"post"},{"content":"A list of blogs/documents I plan to read.\nHarness Engineering\nhttps://www.anthropic.com/engineering/harness-design-long-running-apps https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents https://openai.com/index/harness-engineering/ https://ghuntley.com/ralph/ https://www.reddit.com/r/ClaudeAI/comments/1s9jm0d/i_had_claude_read_every_harness_engineering_guide/ Blogs to help understand how to write prompts for Agents:\nhttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents https://www.anthropic.com/engineering/writing-tools-for-agents https://developers.openai.com/api/docs/guides/prompt-guidance https://developers.openai.com/cookbook/examples/gpt-5/prompt_personalities https://developers.openai.com/cookbook/articles/codex_exec_plans https://platform.claude.com/cookbook/tool-evaluation-tool-evaluation https://www.anthropic.com/research/tracing-thoughts-language-model https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools#best-practices-for-tool-… https://anthropic.skilljar.com/introduction-to-agent-skills Blogs to help learn about building Gaurdrails\nhttps://docs.langchain.com/oss/python/langchain/guardrails https://www.anthropic.com/news/building-safeguards-for-claude https://developers.openai.com/cookbook/topic/guardrails ","date":"25 May 2026","externalUrl":null,"permalink":"/future-reads/","section":"Home Page","summary":"","title":"Future Reads","type":"page"},{"content":"The ReACT framework is one of the most popular ways to build agentic systems today. However, I recently came across an enhanced version that makes the architecture much more scalable and easier to maintain.\nI’ve put together a video to explain the \u0026ldquo;how\u0026rdquo; and \u0026ldquo;why\u0026rdquo; behind this setup, including how to use node-pool architecture to level up your agents.\nWatch the video here: You can also explore the prototype codebase in my GitHub repo:\nView Code on GitHub\n","date":"4 May 2026","externalUrl":null,"permalink":"/post/react-and-node-pool-architectre/","section":"Post","summary":"Learn how to build better agentic systems by enhancing the ReACT framework with a node-pool architecture. This guide covers how to make your AI agents more scalable and maintainable, including a video walkthrough and a GitHub prototype.","title":"ReAct and Node-Pool Architecture","type":"post"},{"content":"","date":"9 March 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"9 March 2026","externalUrl":null,"permalink":"/tags/newlearning/","section":"Tags","summary":"","title":"NewLearning","type":"tags"},{"content":"","date":"9 March 2026","externalUrl":null,"permalink":"/categories/quantum-computing/","section":"Categories","summary":"","title":"Quantum Computing","type":"categories"},{"content":"I completed the first introductory section from Black Opal and it was really great. It explained what a quantum computing system looks like. Seeing the system visually makes a huge difference compared to just imagining things in your head.\nThe other courses are not free. They need a paid membership, so I decided to buy the monthly one. But when I was paying with my credit card, I bumped into an issue. Black Opal tries to make the payment without requesting an OTP, which is a mandatory step in India.\nI found Michael Biercuk, CEO \u0026amp; Founder of Q-CTRL, to be very active on LinkedIn, so I decided to reach out to him. I dropped a message stating the problem, and he replied asking for my email address so his team could handle this situation. I of course thanked him and waited for their email.\nThey were really quick to respond. They emailed me about my preferred payment mode and even offered a 20% discount! But the problem is, I still haven\u0026rsquo;t received the Stripe payment link. I am eagerly waiting for it. This platform looks solid, and I really want to study more about Quantum Computing. Hopefully, I can get this sorted early this week.\nNothing much else this week. Looking forward to the next one.\nThat\u0026rsquo;s it for now, See you next Sunday!\n","date":"9 March 2026","externalUrl":null,"permalink":"/post/qq-week-4-update/","section":"Post","summary":"Week 4 update: Completed Black Opal intro, payment issues with Indian cards, and reaching out to the CEO.","title":"Quantum Computing Week 4 Update","type":"post"},{"content":"","date":"9 March 2026","externalUrl":null,"permalink":"/tags/quantumcomputing/","section":"Tags","summary":"","title":"QuantumComputing","type":"tags"},{"content":"","date":"9 March 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"I have completed the IBM course on \u0026ldquo;Basics of Quantum Information\u0026rdquo;. Last week, I learned about the limitations of quantum computing, teleportation, and superdense coding.\nIn quantum computing, it is impossible to copy data like we do in classical programming languages.\nx = 10 y = x To copy a qubit, we need to read it first. However, reading a qubit changes its state, destroying the original state due to it being in superposition(50/50). Teleportation and Superdense coding are two ways to transfer information, where Entanglement is the core principle.\nTo be honest, I am still finding it hard to grasp everything. I think I only understand about 15% right now. Maybe with another 2-3 months of consistency, I will improve. But I am loving the time I spend on this future technology. It feels like a well-spent hour.\nAfter completing the IBM course, I was looking for what to read next. I found two good sources: Black Opal and the Fundamentals of Quantum Information course by TUDelft.\nI wanted to start with TUDelft\u0026rsquo;s course, but I figured it might need more experience. I felt I might not be able to do justice to it right now, so I decided to do it later.\nI am starting with Black Opal. I browsed the website and it looks really cool. I checked the LinkedIn profiles of the founder and key people, and I loved their vision. So, I decided to go with Black Opal first.\nThat\u0026rsquo;s it for now, See you next Sunday!\n","date":"4 March 2026","externalUrl":null,"permalink":"/post/qq-week-3-update/","section":"Post","summary":"Week 3 update: Completed IBM course, learned about No-Cloning theorem, Teleportation, and Superdense coding. Next stop: Black Opal.","title":"Quantum Computing Week 3 Update","type":"post"},{"content":"I am back. The second week was a little different from week one. To be honest, I did not study all seven days; I had to skip two days due to my cats not letting me sleep and hence sleep deprivation (not another excuse, Akansha!!).\nI started with the IBM course Basics of Quantum Information. I started with the video on the Introduction page, but realised I relate more to the textual content; it helps me focus better. So to make sure I was not missing anything, I decided to watch the video, and after 40 minutes, I moved to the textual content to see if the content was the same as in the video. And it was the same.\nSo I resumed with the textual content. It was a little overwhelming. I learned about ket and bra. A ket is a column vector, whereas a bra is a row vector. I learned about the unitary operation and installed Qiskit, and here is the first piece of code from this journey. I also learned what Jupyter Notebook is, and how to use it inside Visual Studio Code and the browser.\nimport numpy as np from qiskit.visualization import array_to_latex from IPython.display import display ket0 = np.array([[1], [0]]) ket1 = np.array([[0], [1]]) M1 = np.array([[1,1], [0,0]]) M2 = np.array([[1,0], [0,1]]) mull = np.matmul(M1, M2) display(array_to_latex(np.matmul(M1, M2))) from qiskit.quantum_info import Statevector from numpy import sqrt u = Statevector([1 / sqrt(2), 1 / sqrt(2)]) v = Statevector([(1 + 2.0j) / 3, -2 / 3]) w = Statevector([1 / 3, 2 / 3]) display(u.draw(\u0026#34;text\u0026#34;)) display(u.draw(\u0026#34;latex\u0026#34;)) display(u.draw(\u0026#34;latex_source\u0026#34;)) outcome, state = v.measure() print(f\u0026#34;Measured: {outcome}\\nPost-measurement state:\u0026#34;) display(state.draw(\u0026#34;latex\u0026#34;)) from qiskit.visualization import plot_histogram statistics = v.sample_counts(1000) plot_histogram(statistics) I would not say it was the best week learning-wise, but it was good. Now I am looking forward to continuing on this journey and seeing what is in there for me. I plan to read Multiple Systems this week.\nThat\u0026rsquo;s it for now. See you next Sunday!\n","date":"23 February 2026","externalUrl":null,"permalink":"/post/qq-week-2-update/","section":"Post","summary":"Week 2 update: Qiskit installation, understand Ket and Bra vectors, and run my first quantum simulation code using Jupyter Notebook.","title":"Quantum Computing Week 2 Update","type":"post"},{"content":"","date":"15 February 2026","externalUrl":null,"permalink":"/tags/announcement/","section":"Tags","summary":"","title":"Announcement","type":"tags"},{"content":"In 2024, I started my AI journey. I did really well, I learned some great stuff and was enjoying it very much. It’s also worth noting that at that time, while there was a lot of talk around AI, we didn\u0026rsquo;t have the \u0026ldquo;AI slop\u0026rdquo; we see today. The learning journey felt really good. I completed ML courses, built many POCs, and even got featured for implementing RAG with Drupal by The Drupal Times in July 2024. I read many books, ranging from core technical AI books to business AI books.\nI was enjoying my time and even moved into a new role in May 2025, specifically in AI Integrations. But nothing much happened after that. In the industry, AI adoption is still a very big challenge. Only a handful of people are talking about real AI, the rest are just sliding into the \u0026ldquo;AI slop\u0026rdquo; category. Reading more about AI lately doesn\u0026rsquo;t feel like I am learning something futuristic anymore.\nI have always seen myself as a futurist, learning things that might not matter today, but will matter in the future. This is how I prepare myself to be ahead of the crowd, rather than just learning things to save a job. I hate learning out of pressure or the fear of losing work. Thankfully, I have landed a new job thanks to all my work in the AI world. Since I will be working on AI in my official job, I don\u0026rsquo;t want to dedicate my personal time to it. I want my personal time for something truly futuristic.\nQuantum Computing is what I have decided to learn, to see if it’s something I find interesting. If it is, I will go deep. The next three months will be a test period for me and Quantum Computing.\nWhy Quantum Computing? # AI is the present. It’s helping us automate repetitive work through its agentic architecture. It is not perfect at the moment, but it will get better as it merges with engineering best practices. However, running AI at that scale requires a different technology, something that can compute at a much faster rate. Quantum Computing is the potential answer. I don\u0026rsquo;t think it is that far in the future before everyone starts talking about it just like they do for AI now. When that time comes, I want to be ahead of the crowd and understand it better.\nWhat is my Plan? # I am starting this journey slow. Unlike my past habit of going \u0026ldquo;all-in\u0026rdquo; on professional commitments until I do nothing else, I have decided to go slow but be consistent. I will dedicate one hour every day to this new world of Quantum Computing.\nI started on 9 Feb 2026 by reading a very good article by Andy Matuschak and Michael Nielsen titled “Quantum Computing for the Very Curious”. I understood a little about Qubits and different gates like NOT, Hadamard, and CNOT. There was some math involved. I understood some parts, and some I did not. I am not making the same mistake this time of going too deep into prerequisites. Instead, I will learn things as they are mentioned in the material.\nNext, I plan to gain more understanding by completing basic courses from the IBM Quantum Learning platform.\nEvery week on Sunday, I plan to publish an article documenting my journey.\nThat\u0026rsquo;s it for now, See you next Sunday!\n","date":"15 February 2026","externalUrl":null,"permalink":"/post/new-learning-journey/","section":"Post","summary":"Pivoting to something new!","title":"New Learning Journey","type":"post"},{"content":"Last week is going to be a great memory for me. I finally got to attend and present at my First DrupalCon, in Nara. I\u0026rsquo;ve been wanting to go for the last 4-5 years, and though I have been to Japan before, this time it felt completely different because I was there to present.\nThe whole experience was surreal. I met and had dinner with the Acquia team (including Dries), which was a highlight. The sessions were really interesting, covering a wide range of topics, from Drupal Canvas to Recipes. Each session delivered a new experience.\nSince it was my first time presenting at DrupalCon and at an international conference, I was nervous, of course. I didn\u0026rsquo;t have breakfast or a good lunch. My session was at 2:10 PM, so I decided to stick to eating fruits. Once the session was delivered, I ate all I could!\nMy preparation ran into two big challenges right before the session.\nThe Script Problem\nI had been working on a perfect script for weeks, a script for each slide, maintaining the flow between slides with some funny jokes and serious sentences. But when I was preparing, I had doubts: would I be able to see my script while presenting? Will there be a second screen?\nAt the venue, I saw the answer was No. There was no way to see or read the script easily. This became a real challenge.\nTo solve this, I quickly created a Google document with my script and decided to keep it open on my phone. I just needed a short glimpse of the script, and I knew I could handle the rest since I had been practicing and enacting the presentation for weeks. I still needed that quick reference, though. So, the script part was sorted: read it from the phone.\nThe Demo Fail\nThen came the live demo part. I had planned to demo the workflow I created on Activepieces, Drupal, and Slack, but about an hour before the session, I did a dry run and saw Activepieces’s LinkedIn piece throwing a version issue with their LinkedIn API.\nIt\u0026rsquo;s not surprising with live demos, it\u0026rsquo;s usual that they don\u0026rsquo;t work when you want to show them. For a backup, I had recorded a dry run video, but I still really wanted to present it working live, so I gave it a shot. I put on my developer hat and decided to call the LinkedIn API directly within the Activepieces workflow. That also did not work because versioning was a problem with the custom API piece as well. I had tried everything, even went back to old-school way of searching over StackOverflow. But nothing worked.\nSo, I had to stick to the recorded demo.\nThe Session Time\nThe moment came. I connected the projector chord to my laptop and shared the screen. I opened the script on my phone and started the presentation.\nI could feel a little nervousness in my voice, but as I started, I just went with the normal flow. I didn\u0026rsquo;t look at the script at all. I was able to deliver the session without looking down. I covered all parts, demoed the workflow (using the video), and everything went smoothly.\nI was very happy with my presentation. I had some people from my team click some pictures, and though they weren\u0026rsquo;t nice pictures, I could hardly see myself but they were great for memory. Post-presentation, I talked with people, asked them how it went, and they were all very happy with it.\nPost that, I was enjoying like a child. That\u0026rsquo;s how I presented at my first DrupalCon.\nThis was an experience that I will remember forever: presenting at my first international conference.\nDrupalCon Sign Board Picture of Akansha presenting at DrupalCon Team Dinner at DrupalCon Nara Akansha presenting at DrupalCon Side View ","date":"22 November 2025","externalUrl":null,"permalink":"/post/drupal-con-nara-experience/","section":"Post","summary":"Last week is going to be a great memory for me. I finally got to attend and present at my First DrupalCon, in Nara. I’ve been wanting to go for the last 4-5 years, and though I have been to Japan before, this time it felt completely different because I was there to present.","title":"My First DrupalCon: Nara, Japan","type":"post"},{"content":"Hi everyone,\nI\u0026rsquo;m excited to share that I will be presenting at DrupalCon Nara about my learnings from integrating AI into marketing teams. I will be sharing a lot of interesting stuff from my slides, but to make the session more impactful, I wanted to talk about how easy it is to automate workflows.\nThese days, we have great no-code and low-code workflow tools like n8n, Zapier, and Activepieces.\nI decided to go with Activepieces for this demo because it matches the Drupal energy, it is truly open-source. Dries Buytaert also talked about this in his DrupalCon Vienna keynote, so I was excited to try it.\nThe Problem: Why Build a \u0026ldquo;Bot\u0026rdquo;? # In this article, I will describe how you can build a simple bot to solve the infamous problem that comes after you publish your content on your website.\nTo publicize your new article, you then have to work on creating content for different social media channels, like LinkedIn, Twitter, and Mailchimp. This is manual work. It\u0026rsquo;s boring, and it takes time.\nThis is a perfect job for a bot! We can easily automate this with a workflow tool (like Activepieces for our demo).\nBut we have a new problem. We don\u0026rsquo;t want a \u0026ldquo;stupid\u0026rdquo; bot posting just anything to our company\u0026rsquo;s social media. We need a \u0026ldquo;smart\u0026rdquo; bot. We need a \u0026ldquo;human-in-the-loop\u0026rdquo; a real person who must approve the content before it goes live.\nThis automation leaves time for the marketing team to do the important work (reviewing and approving) and lets the bot do the boring work (posting).\nSo, Let\u0026rsquo;s Get Started # Let\u0026rsquo;s go step by step. Here is how we built our smart bot.\nStep 1: Set up the \u0026ldquo;Ear\u0026rdquo; (The Webhook at the Drupal Site) # This is the starting point. Our bot needs an \u0026ldquo;ear\u0026rdquo; to listen for new posts. We use the standard Webhooks module in Drupal for this.\nWe set up a new webhook that triggers after a new \u0026ldquo;Blog Post\u0026rdquo; is saved and published. We configure this webhook to send all the article data (like the title and the full HTML body) to a special URL that Activepieces will give us in the next step.\nStep 2: Build the \u0026ldquo;Brain\u0026rdquo; (The Flow in Activepieces) # This is where all the logic happens. Here is the exact, step-by-step guide to build the bot\u0026rsquo;s brain.\n1. Trigger: Webhook # What it does: This is the first step in your flow. It gets the URL from Step 1. It just waits to \u0026ldquo;hear\u0026rdquo; the data from Drupal. Pro-Tip: Click \u0026ldquo;Test trigger\u0026rdquo; in Activepieces and then publish a test post in Drupal. This will pull in a sample of your data, which makes the next steps much easier. 2. Clean the Data # Piece: Text Helper Action: Remove HTML Why: The data from Drupal is in HTML (like \u0026lt;p\u0026gt;, \u0026lt;h2\u0026gt;, etc.). Our bot\u0026rsquo;s brain (the AI) needs clean text. This step \u0026ldquo;cleans\u0026rdquo; the article. Input: In the \u0026ldquo;Text\u0026rdquo; field, map the body.entity.body[0].value data from Step 1. 3. Ask the AI Assistant for Ideas # Piece: OpenAI Action: Ask GPT-4o (or any model you like) Why: This is our AI assistant. We give it the clean text and ask it to write our social media posts. The most important part is telling it to return the answer in JSON format so the bot can read it. Prompt: Use a prompt like this. You are an expert social media copywriter. Your only job is to return a valid, minified JSON object. Do not add any text before or after the JSON.\nBased on the following article: [Drag the 'text' output from Step 2 here]\nYour JSON output must follow this structure:\n{ \u0026#34;linkedinPost\u0026#34;: \u0026#34;Your professional LinkedIn summary here.\u0026#34;, \u0026#34;twitterPost\u0026#34;: \u0026#34;Your engaging, punchy tweet here with hashtags.\u0026#34; } 4. Understand the AI\u0026rsquo;s Answer # Piece: Code Why: The AI (Step 3) will sometimes wrap its JSON answer in text like ` ``json \u0026hellip; ``` `. This will cause an error. We use a small code step to safely extract only the clean JSON. Input: Create an input called raw_text and map the AI\u0026rsquo;s response (from Step 3) to it. Code: Paste this code into the code box. export const code = async (inputs) =\u0026gt; { const rawText = inputs.raw_text || \u0026#34;\u0026#34;; // This regex finds the JSON between the ```json tags const jsonMatch = rawText.match(/```json\\n([\\sS]*)\\n```/); const jsonString = jsonMatch ? jsonMatch[1] : rawText; try { return JSON.parse(jsonString); } catch (e) { return { error: \u0026#34;Failed to parse JSON from AI\u0026#34; }; } }; The output of this step will be two clean data pills: linkedinPost and twitterPost. 5. Build the \u0026ldquo;Human-in-the-Loop\u0026rdquo; Gate # Piece: Slack Action: Request Approval in a Channel Why: This is the most important step of our smart bot! It pauses the flow and sends a message to a human for review. Message: In the \u0026ldquo;Message\u0026rdquo; field, you can now use the data from Step 4. New post for approval:\nLinkedIn: [Drag the 'linkedinPost' output from Step 4] Twitter: [Drag the 'twitterPost' output from Step 4]\n6. The Bot\u0026rsquo;s Decision-Making # Piece: Route Why: This step checks what button the human clicked in Slack. Route 1: Add a route and set the condition: First Value: approve Condition: (Text) Is Second Value: true The \u0026ldquo;Default\u0026rdquo; path will be our \u0026ldquo;Reject\u0026rdquo; path, where the bot does nothing. 7. The Bot Does the Boring Work # Inside \u0026ldquo;Route 1\u0026rdquo;, add the steps for the bot to do its job: Piece: LinkedIn Action: Create Company Update Text: [Drag the 'linkedinPost' output from Step 4] Piece: Twitter Action: Post Tweet Text: [Drag the 'twitterPost' output from Step 4] You can also add a Slack step here to send a \u0026ldquo;✅ Success!\u0026rdquo; message. The Final Demo # I will run this demo live on stage at DrupalCon Nara.\nLet\u0026rsquo;s see if it works!!\n","date":"14 November 2025","externalUrl":null,"permalink":"/post/demo-with-activepieces/","section":"Post","summary":"A step-by-step guide on how to build a human-in-the-loop social media bot with Drupal, Activepieces, and AI. Automate your posts with a safety check","title":"How I Built a Human-in-the-Loop Social Media Bot with Drupal and Activepieces","type":"post"},{"content":"I spent last week reading various resources, ranging from official documentation from Anthropic, OpenAI, and Microsoft, to Harvard\u0026rsquo;s blog and the Prompt Engineering Guide by DAIR.AI, and I started to see a pattern. Interacting with Large Language Models (LLMs) isn\u0026rsquo;t just about asking questions; it\u0026rsquo;s about providing a structured input that predictably yields a desired output. It\u0026rsquo;s less like a conversation and more like a well-formed API call.\nFor anyone in the tech space looking to get more out of these models, the quality of your output is a direct function of the quality of your input. Here, I\u0026rsquo;ve converted my learnings into a practical framework, moving from basic commands to more advanced mental models for tackling complex tasks.\nPro Tip: This pattern can be applied to any field, not just development.\nLevel 1: The Default Approach: Zero-Shot Prompting # Most interactions with an LLM start here. A Zero-Shot Prompt is essentially a direct command without any prior context or examples. You ask for something and trust the model\u0026rsquo;s pre-trained knowledge to figure it out.\nA typical Zero-Shot prompt:\nGenerate a Python function that calculates the factorial of a number.\nZero-shot prompt output screenshot (click to enlarge) The model produced a correct function. But for anything more specific, like requiring a specific style, error handling, or documentation, this approach is a pure hit-n-trail. The output is functional but unrefined because the instruction lacked specificity.\nLevel 2: The Practical Upgrade: Structured, Example-Driven Prompting # To get reliable and high-quality results, we need to provide the model with a better specification. This involves two key upgrades: providing a structured brief and giving it an example to follow.\n1. The Structured Brief (The C-R-A-F-T Framework) # I\u0026rsquo;ve found it useful to structure my prompts using a mental model I call C-R-A-F-T. It ensures I provide all the necessary parameters for the \u0026ldquo;API call\u0026rdquo; to the LLM.\nC - Context: The background and scope of the task. R - Role: The persona the model should adopt (e.g., \u0026ldquo;Act as a senior software architect\u0026rdquo;). A - Action: The specific verb for the task (e.g., \u0026ldquo;Refactor,\u0026rdquo; \u0026ldquo;Generate,\u0026rdquo; \u0026ldquo;Summarize,\u0026rdquo; \u0026ldquo;Analyze\u0026rdquo;). F - Format: The desired output structure (e.g., \u0026ldquo;JSON format,\u0026rdquo; \u0026ldquo;a markdown table,\u0026rdquo; \u0026ldquo;a bulleted list\u0026rdquo;). T - Target: The intended audience for the output. 2. Example-Driven Guidance (Few-Shot Prompting) # A Few-Shot Prompt is the most effective way to guide the model\u0026rsquo;s output style. By providing one or more examples, you give the model a concrete pattern to replicate.\nPutting it all together for a technical task:\n(Role) Act as a Python developer specializing in clean, readable code. (Context) I am writing a utility script and need a function to fetch data from a public API. (Action) Write a Python function that takes a URL as an argument and returns the JSON response. (Format) The function must include a docstring explaining its purpose, parameters, and return value. It should also include basic error handling for network requests.\n(Few-Shot Example) Here is an example of my preferred coding style:\ndef add(a: int, b: int) -\u0026gt; int: \u0026#34;\u0026#34;\u0026#34;Adds two integers together. Args: a: The first integer. b: The second integer. Returns: The sum of the two integers. \u0026#34;\u0026#34;\u0026#34; return a + b This combined approach moves from a vague request to a precise specification. The resulting code will not only be functional but will also match your required standards for documentation and structure.\nC-R-A-F-T + Few-Shot prompt output screenshot (click to enlarge) Level 3: Advanced Mental Models for Complex Tasks # For tasks that require reasoning or creativity, we need to guide the model\u0026rsquo;s \u0026ldquo;thought process.\u0026rdquo;\nFor Logical Reasoning: Chain of Thought (CoT) # When a task involves multiple steps, you can use Chain of Thought (CoT) prompting by simply instructing the model to \u0026ldquo;think step-by-step.\u0026rdquo; This forces it to externalize its reasoning process, which often leads to more accurate and logical conclusions. It\u0026rsquo;s the difference between asking for a final answer and asking the model to show its work, the latter is better right? Let\u0026rsquo;s understand with the help of an example:\nThe Goal: Debug a complex configuration issue.\nFirst, the vague prompt that gives a less helpful answer:\nWhy is my Docker container failing to connect to the database?\nNow, the far more effective Chain of Thought prompt:\nI\u0026rsquo;m debugging a Docker networking issue. My application container can\u0026rsquo;t connect to my database container. Thinking step-by-step, list the potential causes and suggest a command to verify each one.\nThis prompt yields a structured, actionable checklist instead of a single guess.\nChain of thought prompt output screenshot (click to enlarge) For Strategic Exploration: Tree of Thought (ToT) # For tasks where there isn\u0026rsquo;t one right answer, like system design or technical strategy, the Tree of Thought (ToT) model is incredibly powerful. You guide the LLM to explore multiple independent lines of reasoning (\u0026ldquo;branches\u0026rdquo;), evaluate their pros and cons, and then synthesize a final recommendation.\nThe Goal: Choose a database for a new application.\nTree of Thought Prompt:\nAct as a senior software architect. I am designing a new social media application.\nFirst, propose three different database options: one SQL, one NoSQL (Document), and one NoSQL (Graph). Next, for each option, briefly analyze its pros and cons specifically for a social media application\u0026rsquo;s data model (e.g., user profiles, posts, social connections). Finally, conclude with a recommendation for which database to start with and provide a brief justification. This approach moves the LLM from being a simple information retriever to a powerful analysis and reasoning partner.\nTree of thought prompt output screenshot (click to enlarge) Cheat Sheet # Technique Primary Use Case Zero-Shot Quick idea generation and baseline outputs. C-R-A-F-T + Few-Shot Creating high quality, structured content that matches a specific style. Chain of Thought (CoT) Solving logical problems, debugging, and creating step-by-step plans. Tree of Thought (ToT) Brainstorming multiple strategies, exploring creative paths, and receiving a final recommendation. Final Takeaway # My biggest takeaway from this journey is that the real skill lies in moving from simply \u0026lsquo;prompting\u0026rsquo; to actual \u0026rsquo;engineering\u0026rsquo;. Treating my interactions with an LLM less like a conversation and more like a well-structured API call was the mental shift that made everything click.\nThe better the spec, the better the output. It\u0026rsquo;s a principle every developer already understands, just applied to a new and powerful context. The initial effort to be more structured is what turns a fascinating tool into a truly reliable one.\n","date":"10 September 2025","externalUrl":null,"permalink":"/post/hack-to-effecting-prompts/","section":"Post","summary":"In this post, I’m sharing my practical framework for crafting better prompts, showing how to move from simple commands to advanced techniques that solve complex problems.","title":"A Practical Framework for Writing Effective Prompts","type":"post"},{"content":"When I started reading this book on AI, it felt very different from the usual ones. Most books i have read focus on how AI works, the code, models, and tools. But this one talked about something much more practical: how AI can help businesses and solve real-world problems.\nWhat is AI, Really? # The book begins with the basics. It explains that AI isn\u0026rsquo;t just a fancy tech buzzword. It’s a tool that can help companies make smarter decisions and work more efficiently.\nClearing Up Common Myths About AI # One of the most interesting parts of the book was where it talked about common myths that people believe about AI.\nMyth 1: “AI will take all our jobs.”\nAI can do some tasks faster, but it won’t replace humans completely. For example, in customer service, AI can answer simple questions, but people are still needed for more complex conversations. Myth 2: “AI is 99% accurate.”\nAI is only as good as the data it learns from. If the data is biased or wrong, the AI will also make mistakes. Myth 3: “AI gives instant results.”\nThat’s not true. A good example is self-driving cars. In 2016, experts said we’d have 10 million driverless cars by 2020. Companies like GM, Toyota, Waymo, and even Elon Musk made big promises. But even now, in 2025, fully driverless cars aren’t common.\nAnother great example is Google Search. Google started in 1996, but it took many years to improve:\nIn 2012, it introduced the Knowledge Graph to better understand what people are searching for. In 2015, it added RankBrain, an AI-based system to improve search results. By 2017, search became even richer with news, videos, and more. These examples show that real progress takes time.\nMyth 4: “AI is less biased than humans.”\nNot always. AI can learn biases from data. For instance, the COMPAS algorithm used in U.S. courts wrongly labeled many Black defendants as high-risk. This showed that AI can make unfair decisions if trained on biased data. Where AI Can Be Used in Business # The book then talks about how AI can help in many areas of a company:\nCustomer Service Human Resources Sales Marketing IT Support Manufacturing A great example is Amazon’s recommendation system. It suggests products based on your behavior, and that brings in about one-third of their revenue!\nIt also shared the story of Google’s People Analytics Team.\nThey used data to learn what makes a good manager. Surprisingly, it wasn’t technical skills. It was about being a good coach, empowring and avoiding micromanagement.\nProcess to Build an HI-AI System # The book shares a clear process that companies can follow to build High Impact AI tools.\nUnderstand the problem Collect and clean data Build the AI model Test it Launch it Monitor and improve it over time This helps make sure the AI is useful and keeps getting better.\nThe B-CIDS Framework: Is Your Company Ready for AI? # In Chapter 8, the book introduces five key things every business needs to be ready for AI. It\u0026rsquo;s called B-CIDS:\nBudget – Is there money to support the project? Culture – Do people in the company believe in data and AI? Infrastructure – Do you have the tools and systems needed? Data – Are you collecting and organizing your data properly? Skills – Do your teams have the right knowledge to work with AI? Each pillar comes with questions like:\nAre you storing and logging your data? Are old paper records being digitized? Are leaders comfortable with using data to make decisions? One of the best examples from the book was the story of Blockbuster and Netflix. Blockbuster had money and a big brand, but it failed to change with the times. Netflix, on the other hand, used data and technology to offer a better experience. The lesson? You don’t need the most advanced AI — you just need to be open to change and use data to make better decisions.\nHow to Find the Right AI Use Cases # The book ends with a very useful framework to help businesses find where AI can make the most impact. There are two ways to spot AI opportunities:\nProactive Discovery – Look for slow, manual, or repeated tasks in the business. Organic Discovery – Take a big problem and break it down to see if AI can help with any part of it. You can then group the problems as:\nOld problems still done manually Problems already handled by software, but not well New problems with no current solution To decide if AI is the right tool, ask:\nIs the problem hard to solve with rules? Is it something people spend a lot of time on? Is good data available? Are current tools not working well? Final Thoughts # I really enjoyed this book. It helped me understand that AI isn’t the answer to everything — and that’s okay. What matters more is having the right setup, the right mindset, and a clear plan to use AI in a meaningful way.\nIf you’re looking to understand how AI can solve real-world problems, not just from a technical point of view, but from a business lens. I highly recommend reading this book. It’s practical, easy to follow, and full of examples that will change the way you think about AI in business.\n","date":"29 May 2025","externalUrl":null,"permalink":"/post/business-case-for-ai-book-take-away/","section":"Post","summary":"A practical summary of key takeaways from a business-focused AI book. It explores real-world use cases, debunks common myths, and explains how companies can prepare for and apply AI effectively using simple frameworks like B-CIDS. Ideal for anyone looking to understand AI beyond just code and models.","title":"The Business Case for AI: My Takey","type":"post"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/tags/ai/","section":"Tags","summary":"","title":"Ai","type":"tags"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/categories/artificial-intelligence/","section":"Categories","summary":"","title":"Artificial-Intelligence","type":"categories"},{"content":"I’ve always found it fascinating how Google Images can find visually similar images almost instantly. It feels magical — you upload one photo, and in seconds, it understands and matches it to millions of others. This weekend, I wanted to explore: Could I build a tiny, minimal version of that, using only open-source tools?\nChoosing the Tools # After some research, I decided to experiment with two open-source projects:\nCLIP (by OpenAI): a model that converts images and text into embeddings, placing them into the same vector space where they can be meaningfully compared.\nFAISS (by Meta): a library designed for fast similarity search over large collections of embeddings, helping you find the nearest matches efficiently. Both seemed lightweight enough for a small project, yet powerful enough to give meaningful results.\nWhat I Built # Here’s a short demo of the image similarity tool I created:\nThe tool lets you:\nUpload images to create a searchable index Search by uploading another image Or even search by typing a text query Behind the scenes, CLIP generates embeddings, and FAISS efficiently finds the nearest matches.\nYou can find the complete code here: GitHub Repository\nA Quick Look at How It Works # When a user uploads an image, I convert it into a CLIP embedding. I store these embeddings in FAISS for fast retrieval. During search, the uploaded image or text query is embedded and compared against the stored vectors. The closest matches are shown back to the user. Here’s the rough architecture I followed:\nThe app itself runs on Streamlit, keeping it simple and easy to experiment with.\nWhat I Learned # Working with CLIP was a great experience. It’s impressive how it generalizes, being able to search images using text without any retraining felt almost like cheating.\nHowever, I also ran into a few limitations:\nFAISS always returns top K results, even if no truly close match exists. With a small index, this can sometimes return completely unrelated images as “similar.” CLIP embeddings are semantic, not instance-specific. For example, two photos of the same cat in different poses don’t necessarily come close in embedding space. CLIP understands the “idea” of a cat, but not necessarily the identity of this cat. Handling file uploads in Streamlit needed a bit of care to avoid processing the same file multiple times when switching tabs. These are all natural limitations of the tools — and understanding them was part of the fun.\nReflections # This project reminded me how much can be achieved with simple, well-built tools. I didn’t train a single model. I didn’t build a database from scratch. I just connected two smart pieces — CLIP and FAISS — and a lot of the magic simply happened. It also made me think: Sometimes, creativity isn’t about building everything yourself, but knowing how to connect things meaningfully.\nWhat’s Next? # This experiment opened up a few ideas I’d like to explore further:\nFine-tuning a model for better instance-specific matching Experimenting with more advanced search methods (e.g., filtering by confidence score) Building a small backend to make this tool scalable beyond a weekend project There’s a lot more to do if I want to make this production-grade. But for now, I’m happy with where this experiment landed.\nThanks for reading! # If you have thoughts, ideas, or just want to geek out about embeddings, feel free to connect!\n","date":"28 April 2025","externalUrl":null,"permalink":"/post/image-similarity-tool/","section":"Post","summary":"I built a small project to explore how machines understand and connect images. Using two open-source tools: CLIP (by OpenAI) and FAISS (by Meta). I created a simple image similarity search engine.","title":"Building an Image Similarity Search Using CLIP and FAISS","type":"post"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/tags/embedding/","section":"Tags","summary":"","title":"Embedding","type":"tags"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/categories/machine-learning/","section":"Categories","summary":"","title":"Machine-Learning","type":"categories"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/tags/ml/","section":"Tags","summary":"","title":"Ml","type":"tags"},{"content":"","date":"28 April 2025","externalUrl":null,"permalink":"/tags/vector-space/","section":"Tags","summary":"","title":"Vector-Space","type":"tags"},{"content":"Hello everyone! Continuing with my journey of exploring \u0026ldquo;Clean Code\u0026rdquo; by Robert C. Martin, I am back with my thoughts on the chapter about Functions. This blog is a continuation of my series on Clean Code insights. If you missed the first post, you can find it here.\nSo, let\u0026rsquo;s dive into some key lessons about writing clean and effective functions.\nFunctions Should Be Small: Functions should be small, doing only one thing and doing it well, as they’re easier to read, understand, and maintain. If you notice sections in a function, it’s a sign it’s doing too much—break it into smaller, focused functions instead.\nOrder of Functions in a Class: While there isn’t a strict rule, functions should ideally be written in the same order they are called. This makes reading the code feel natural, like reading a newspaper article.\nDescriptive Names: A function’s name should clearly say what it does. It doesn’t matter if the name is long; what matters is clarity. A well-named function saves time and confusion later.\nAs mentioned in the book, \u0026ldquo;Don’t be afraid to make a name long. A long descriptive name is better than a short enigmatic name.\u0026rdquo;\nKeep Function Arguments Minimal: The fewer arguments a function has, the better.\nBest: 0 arguments Good: 1 or 2 arguments Okay: 3 arguments Avoid: More than 3 arguments (it gets confusing). Avoid Flag Arguments (true/false): If a function takes a boolean flag, it’s a sign it’s doing more than one thing. Instead, split it into separate functions.\n# Not recommended render(bool isMale): if isMale: print(\u0026#34;Washroom is on the right\u0026#34;) else: print(\u0026#34;Washroom is on the left\u0026#34;) # Better renderMensWashroomDirection(): print(\u0026#34;Washroom is on the right\u0026#34;) renderWomensWashroomDirection(): print(\u0026#34;Washroom is on the left\u0026#34;) Use Classes for Many Arguments: If a function requires too many arguments, consider wrapping the arguments in their own class. For example:\n# Too many arguments makeCircle(double x, double y, double radius) # Better makeCircle(Point center, double radius) Order of Arguments: Be mindful of the order of arguments. A descriptive name for the function can help avoid confusion about the argument order. Example:\n# Confusing sendEmail(string message, string recipient, string subject) # Better sendEmailToRecipientWithSubjectAndMessage(string recipient, string subject, string message) In the better version, the function name itself makes the expected order clear, reducing the chances of mixing up arguments.\nWorried About Remembering Long Function Names? Don’t be! Modern code editors auto-complete function names, so you don’t have to type them fully every time. It’s better to be clear than to use short names that cause confusion.\nDelete Unused Functions: If a function is never called, remove it to keep the code clean—no worries, version control (Git) can always bring it back if needed.\nError Handling: Functions should do one thing, and if a function is meant to handle errors, it should do only that and nothing else.\nSingle Responsibility Principle One Function = One Task: A function should either do something or answer something, but not both. For example:\n// Not recommended public boolean set(String attribute, String value) { if (attribute.equals(\u0026#34;username\u0026#34;)) { this.username = value; return true; } return false; } This function sets a value and returns a status, which mixes two responsibilities. Remember the first point: If a function needs to do more than one thing, split it into separate functions.\nFinal Thoughts # Writing clean functions isn’t just about following rules—it’s about making your code easier to read, maintain, and scale. Small, well-named, and focused functions reduce complexity and make debugging easy.\nThe key takeaway? Keep it simple, make it clear, and let your code speak for itself.\nFuel My Work # If you’ve found something here helpful, consider buying me a coffee (or helping me keep this server running)! ☕ Support me HERE\nNeed Guidance or Mentorship? # I’m happy to help! Whether it\u0026rsquo;s coding, career advice, or tech insights, feel free to reach out to me on Topmate: https://topmate.io/saxenaakansha30\n","date":"5 February 2025","externalUrl":null,"permalink":"/post/clean-code-function/","section":"Post","summary":"Continuing with my journey of exploring Clean Code by Robert C. Martin, I am back with my thoughts on the chapter about Functions. This blog is a continuation of my series on Clean Code insights.","title":"Why Clean Code Matters: Insights on Function","type":"post"},{"content":" Clean Code by Robert C. Martin I recently finished reading Clean Code by Robert C. Martin, and I loved many points that I would implement in my approach to coding. Over the next few posts, I’ll share some key takeaways from the book, starting with a topic that’s often overlooked: comments.\nWhile comments can be useful, they can also hide bad coding practices. Here are some Do’s and Don’ts for writing good comments, inspired by the book:\nDon’t Explain Bad Code Comments don’t make up for bad code. If your code is hard to understand, fix the code instead of adding comments to explain it.\nDon’t State the Obvious Comments like this are not helpful:\n# Increment the counter counter += 1 Don’t Use Comments as a Journal We often leave notes in the code about the author, reasons for changes, etc. Avoid doing this. Use version control tools like Git to track the history instead.\nDon’t Leave Commented-Out Code Delete code you’re not using. Version control systems can always bring it back if needed.\nInvest in Quality Well-thought-out comments save time for others and yourself. Don’t ramble; be brief but purposeful. If you want someone to invest their time in reading your comment, make it worth their effort.\nFinal Thoughts # Comments should help, not confuse. They are not a replacement for writing clean and clear code. Remember:\n“Good code is its own best documentation.”\n– Robert C. Martin These are just a few takeaways from Clean Code about comments. I’ll share more in upcoming posts, including tips on writing better functions.\nWhat’s your approach to writing comments? I’d love to hear your thoughts!\nFuel My Work # If you’ve found something here helpful, consider buying me a coffee (or helping me keep this server running)! ☕ Support me HERE\nNeed Guidance or Mentorship? # I’m happy to help! Whether it\u0026rsquo;s coding, career advice, or tech insights, feel free to reach out to me on Topmate: https://topmate.io/saxenaakansha30\n","date":"1 February 2025","externalUrl":null,"permalink":"/post/clean-code-comments/","section":"Post","summary":"I recently finished reading Clean Code by Robert C. Martin, and I loved many points that I would implement in my approach to coding. Over the next few posts, I’ll share some key takeaways from the book, starting with a topic that’s often overlooked: comments.","title":"Why Clean Code Matters: Insights on Comments","type":"post"},{"content":"Hey there, awesome visitor! 👋\nMaintaining this website, writing code, and building exciting tech demos takes time, effort, and, of course, coffee!\nIf my work has helped you in any way—whether through tutorials, open-source projects, or insightful tech posts—consider supporting my work. Your contribution helps me:\nKeep this site up and running. Build and share more in-depth tutorials. Explore and experiment with new technologies. ☕ Buy me a coffee: https://buymeacoffee.com/saxenaakan8\n💰 Prefer UPI? It’s simple! Just set an amount and hit “Generate QR Code” to support my work instantly.\nEvery coffee fuels more tech experiments, better content, and new innovations. Thanks for being part of this journey! 🚀\nEnter Amount: Generate QR Code\n","date":"9 December 2024","externalUrl":null,"permalink":"/fuel-my-work/","section":"Home Page","summary":"","title":"Fuel My Work","type":"page"},{"content":"","date":"5 December 2024","externalUrl":null,"permalink":"/future/","section":"Futures","summary":"","title":"Futures","type":"future"},{"content":" Phase 1: Core AI/ML Foundations (0–4 Months) # Timeline: 20 Dec 2024 to 20 April 2025\nGoals: # Build foundational skills in AI/ML through projects and structured learning. Transition from day-long challenges to in-depth weeklong or biweekly projects. Action Plan: # Complete the 4-Month AI/ML Challenge**:\nFocus on Computer Vision, Recommender Systems, and Reinforcement Learning. Develop end-to-end projects with deployment. Leverage Web Development Experience:\nPractice deploying ML models as APIs using FastAPI. Work with cloud platforms like AWS, GCP, or Azure to host models. Phase 2: Build a Portfolio and Real-World Expertise (4–9 Months) # Note: This period will be a little more hectic because of personal commitments, so keeping that in mind.\nTimeline: 1 May 2025 to 30 Sep 2025\nGoals: # Transition from learning to solving real-world problems. Action Plan: # Capstone Project: Build an end-to-end project with real-world relevance. Examples: Computer Vision: Real-time object detection for smart cameras. NLP: AI-based document summarizer for businesses. Recommender System: Personalized book or movie recommendation engine. Phase 3: University Preparation (9–12 Months) # Timeline: 1 October 2025 to 30 December 2025\nGoals: # Prepare for applications to top European master’s programs in Machine Learning. Build a competitive profile with projects, test scores, and compelling documents. Action Plan: # Identify Target Universities and Programs: Research top European universities offering ML-related programs: Review admission requirements and deadlines. Refine your portfolio to showcase your expertise in AI/ML. Focus on projects relevant to the program’s research areas (e.g., computer vision, generative AI). Create a dedicated portfolio on personal website summarizing the projects. ","date":"5 December 2024","externalUrl":null,"permalink":"/future/masters/","section":"Futures","summary":"","title":"Preparation for Masters","type":"future"},{"content":"After completing the 30 Days ML and DL challenges, the time has come to shift focus from day-long tasks to longer, more complex projects where I can dive deeper into each topic.\nI have designed this 4-month learning plan (of course with the help of ChatGPT) to help in going deeper in the AI and ML world with intensive weeklong to biweekly projects. It includes practical, hands-on tasks, cutting-edge topics, and real-world applications. The plan focuses on:\nComputer Vision Projects to deepen expertise in image processing. Recommender Systems and Reinforcement Learning for practical and industry-ready skills. Generative and Transformer Models for exploring experimental, cutting-edge topics. Capstone Projects to combine skills into impactful applications. Month 1: Computer Vision Focus 🎯 # Week(s) Project Title Key Focus Areas 1–2 Build an Advanced Object Detection System - YOLOv5 or Faster R-CNN- Custom dataset for vehicle or pedestrian detection 3–4 Image Segmentation with U-Net - Semantic segmentation on medical or urban datasets Month 2: Generative Models and Transformers ✨ # Week(s) Project Title Key Focus Areas 5–6 Style Transfer and Artistic AI - Neural Style Transfer or CycleGAN for artistic applications 7–8 Build a Custom Vision Transformer (ViT) - Vision Transformer for image classification Month 3: Recommender Systems and Reinforcement Learning 🎮 # Week(s) Project Title Key Focus Areas 9–10 Build a Personalized Recommender System - Use embeddings from transformers or collaborative filtering- Deploy as a web app 11–12 Reinforcement Learning for Game AI - Train an RL agent for a game like Pong or CartPole- Implement DQN or PPO Month 4: Capstone and Cutting-Edge AI Projects ✨ # Week(s) Project Title Key Focus Areas 13–14 Generative Models with StyleGAN2 - StyleGAN2 for high-quality image synthesis 15–16 Capstone Project: Your Vision - Combine techniques to build a meaningful application- Example: Smart surveillance, AI art app How to Approach This Plan 📅 # Time Commitment: Each project is designed for 1–2 hours daily over 1–2 weeks. Daily Workflow: Week 1: Day 1–2: Research (papers, blogs, and tutorials) and dataset setup. Day 3–4: Design the architecture or adapt a pre-trained model. Day 5: Train the base model (test on small data subsets first). Day 6–7: Evaluate and analyze initial results. Week 2: Day 8–9: Improve performance (hyperparameter tuning, data augmentation). Day 10: Test variations or implement additional features. Day 11–12: Finalize model and document results (visualizations, write-up). Day 13–14: Make YouTube video on the project. Follow Along! # Follow my journey as I dive deeper into AI/ML, and feel free to reach out or comment with questions, suggestions, or your own project ideas! Don’t forget to check out my YouTube channel for regular updates.\n","date":"2 December 2024","externalUrl":null,"permalink":"/post/ml-weeks-long-projects/","section":"Post","summary":"A detailed 4-month plan to learn AI/ML with projects, focusing on Computer Vision, Recommender Systems, Reinforcement Learning, and Generative Models.","title":"4-Month Advanced AI/ML Learning Plan: Biweekly Projects + Advanced Skills Development","type":"post"},{"content":"","date":"2 December 2024","externalUrl":null,"permalink":"/tags/deep_learning/","section":"Tags","summary":"","title":"Deep_learning","type":"tags"},{"content":"","date":"30 November 2024","externalUrl":null,"permalink":"/challenge/","section":"Challenges","summary":"","title":"Challenges","type":"challenge"},{"content":"Today marks the final day of the 30 Days 30 Machine Learning Projects Challenge. I completed the SimCLR self-supervised learning framework by training it on the CIFAR-10 dataset and evaluating the learned representations using a simple classifier.\nSimCLR Recap from Day 29 # In Day 29, I implemented the foundational components of SimCLR:\nData Augmentation: Generated diverse views of the same image. Encoder Network: Extracted meaningful features using a ResNet-50 backbone. Projection Head: Mapped the features to a lower-dimensional space for contrastive learning. Contrastive Loss: Learned representations by pulling augmented views of the same image closer and pushing others apart. Day 30: Training and Evaluation # Today, I focused on:\nTraining SimCLR to learn representations. Evaluating the learned representations using a simple classifier. Step 1: Dataset Preparation for Training # The CIFAR-10 dataset was prepared by:\nNormalization: Rescaled pixel values to the range [-1, 1]. Augmentation Pipeline: Applied random crops, flips, and color distortions to generate diverse views of the data. Batching and Prefetching: Optimized the dataset pipeline for training efficiency. AUTOTUNE = tf.data.AUTOTUNE BATCH_SIZE = 128 BUFFER_SIZE = 10000 def prepare_data(x_data): dataset = tf.data.Dataset.from_tensor_slices(x_data) dataset = dataset.shuffle(BUFFER_SIZE).map(data_augment, num_parallel_calls=AUTOTUNE) dataset = dataset.batch(BATCH_SIZE).prefetch(AUTOTUNE) return dataset dataset = prepare_data(X_data) Step 2: Training SimCLR # The SimCLR framework was trained using the contrastive loss function:\nTwo Augmented Views: For each batch, two augmented views of the same image were generated. Forward Pass: Both views were passed through the encoder and projection head to generate latent representations. Contrastive Loss: Encouraged the model to pull similar views closer in the latent space while pushing others apart. Gradient Updates: The model parameters were updated using the Adam optimizer. optimizer = Adam(learning_rate=0.0003) # Training Steps @tf.function def train_steps(batch): # Generate two augmented views of the image augmented_1 = tf.map_fn(data_augment, batch) augmented_2 = tf.map_fn(data_augment, batch) with tf.GradientTape() as tape: # Encoder and project both z_i = project_head(augmented_1, training=True) z_j = project_head(augmented_2, training=True) # Calculate the contrastive loss loss = contrastive_loss(z_i, z_j) # Apply the gradient gradients = tape.gradient(loss, project_head.trainable_variables) optimizer.apply_gradients(zip(gradients, project_head.trainable_variables)) return loss The model was trained for 10 epochs, and loss was monitored for each epoch.\nEPOCHS = 10 for epoch in range(EPOCHS): epoch_loss_avg = tf.keras.metrics.Mean() for batch in dataset: loss = train_steps(batch) epoch_loss_avg.update_state(loss) print(f\u0026#34;Epoch: {epoch + 1}, loss: {epoch_loss_avg.result().numpy()}\u0026#34;) Step 3: Evaluating Representations with a Classifier # To evaluate the representations learned by SimCLR:\nEncoder Freezing: The encoder was frozen to prevent further updates. Simple Classifier: A dense layer with a softmax activation was added on top of the encoder. This layer mapped the learned features to the 10 CIFAR-10 classes. Training the Classifier: The CIFAR-10 dataset was split into training (80%) and validation (20%) subsets. The classifier was trained for 5 epochs using the frozen encoder features. # Evaluate the Model encoder.trainable = False # Create a simple classifier classifier = tf.keras.Sequential( encoder, Dense(10, activation=\u0026#39;softmax\u0026#39;) ) classifier.compile( optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;] ) # Split into training (80%) and validation (20%) dataset X_train, X_val = train_test_split(X_data, test_size=0.2, random_state=42) # Train the classifier classifier.fit( X_train, y_train, validation_data=(X_val, y_val), epochs=5, batch_size=128 ) This concludes the 30 Days 30 Machine Learning Projects Challenge.\nVideo # ","date":"30 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_30/","section":"Challenges","summary":"Today marks the final day of the 30 Days 30 Machine Learning Projects Challenge. I completed the SimCLR self-supervised learning framework by training it on the CIFAR-10 dataset and evaluating the learned representations using a simple classifier.","title":"Day 30: SimCLR - Self-Supervised Learning Part 2 and Classifier Training","type":"challenge"},{"content":"Today, I started working on SimCLR, a self-supervised learning approach for representation learning. Unlike traditional supervised learning, SimCLR does not require labels for training. Instead, it leverages contrastive learning to learn meaningful representations by comparing augmented versions of the same image.\nProblem Statement # The task was to implement the first part of the SimCLR framework, which involves:\nLoading the dataset and preparing it for self-supervised learning. Defining a robust data augmentation pipeline. Building the encoder network for feature extraction. Adding a projection head to map features into a lower-dimensional latent space. Implementing the contrastive loss function, which is the core of SimCLR. Dataset # The CIFAR-10 dataset was used for this project. It contains:\n50,000 training images and 10,000 test images, each of size 32x32. In self-supervised learning, the labels are not used for training. Preprocessing:\nThe pixel values of the images were normalized to the range [-1, 1] to align with the requirements of the tanh activation function used in the network. Code: # # Problem: Work on SimCLR self-supervised learning import tensorflow as tf from tensorflow.keras.layers import Dense, GlobalAveragePooling2D from tensorflow.keras.applications import ResNet50 import numpy as np import matplotlib.pyplot as plt from tensorflow.keras.datasets import cifar10 # Load the CIFAR-10 dataset (X_train, _), (X_val, _) = cifar10.load_data() # Combine the training and test data. # In Self super-vised training technique we dont need label for tranining X_data = np.concatenate((X_train, X_val), axis=0) # Normalize the pixel values between -1 and 1 (Helps with `tanh` activation function) X_data = (X_data.astype(\u0026#39;float32\u0026#39;) / 127.5) - 1.0 # Define the Augmentation def data_augment(image): # Random crop and resize image = tf.image.random_crop(image, size=[28, 28, 3]) image = tf.image.resize(image, (32, 32)) # Random flip (Left-Right) imagee = tf.image.random_flip_left_right(image) # Color distortion image = tf.image.random_brightness(image, max_delta=0.5) imagee = tf.image.random_contrast(image, lower=0.1, upper=0.9) return image # Visualize some autmented images fig, axs = plt.subplots(1, 4, figsize=(10, 3)) for i in range(4): image = data_augment(X_data[np.random.randint(len(X_data))]) axs[i].imshow((image + 1) / 2) # Rescale it back to 0 and 1 axs[i].axis(\u0026#39;off\u0026#39;) plt.show() # Set up the Base Network (Encoder) def create_encoder(): base_model = ResNet50(include_top=False, weights=\u0026#39;imagenet\u0026#39;, input_shape=(32, 32, 3)) base_model.trainable = True # We want to train the base model from scratch inputs = tf.keras.Input(shape=(32, 32, 3)) x = base_model(inputs, trainable=True) x = GlobalAveragePooling2D()(x) return tf.keras.Model(inputs, x) encoder = create_encoder() encoder.summary() # Create Project Head def create_project_head(encoder): inputs = encoder.input x = encoder.output x = Dense(256, activation=\u0026#39;relu\u0026#39;)(x) output = Dense(128)(x) # Final layer will used for contrastive learning return tf.keras.Model(inputs, output) project_head = create_project_head(encoder) project_head.summary() # Define the Contrastive Loss def contrastive_loss(z_i, z_j, temperature=0.5): # Normalize the two vectors z_i = tf.math.l2_normalize(z_i, axis=1) z_j = tf.math.l2_normalize(z_j, axis=2) # Compute cosine scores similarity_matrix = tf.matmul(z_i, z_j, transpose_b=True) logits = similarity_matrix / temperature # Labels and indices of the positive pair batch_size = tf.shape(z_i)[0] labels = tf.range(batch_size) # Calculate the cross entropy loss loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=labels, logits=logits) return tf.reduce_mean(loss) Step 1: Data Augmentation # A robust data augmentation pipeline was defined to create different views of the same image:\nRandom Cropping and Resizing: Introduces spatial variation. Random Flipping: Makes the model invariant to left-right flips. Color Distortion: Randomly adjusts brightness and contrast to encourage the model to focus on semantic features rather than colors. Here’s a visualization of some augmented images:\nStep 2: Encoder Network # A ResNet-50 model was used as the encoder network:\nPre-trained Weights: The imagenet weights were loaded for initialization. Global Average Pooling: The features extracted by ResNet were pooled to reduce their dimensionality. The encoder serves as the backbone of SimCLR, extracting meaningful representations from the input images.\nStep 3: Projection Head # A projection head was added on top of the encoder:\nDense Layers: A hidden layer with 256 neurons and ReLU activation. A final output layer with 128 neurons for contrastive learning. This projection head maps the features from the encoder to a lower-dimensional latent space where contrastive loss is applied.\nStep 4: Contrastive Loss Function # The contrastive loss encourages the model to:\nBring augmented views of the same image closer together in the latent space. Push apart representations of different images. Next Steps # Training the SimCLR Framework:\nImplement the training loop using the encoder, projection head, and contrastive loss. Evaluate the representations learned by SimCLR. Visualization:\nPlot the loss curve during training. Use a t-SNE plot to visualize the learned representations in 2D space. Video # ","date":"29 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_29/","section":"Challenges","summary":"Today, I started working on SimCLR, a self-supervised learning approach for representation learning. Unlike traditional supervised learning, SimCLR does not require labels for training. Instead, it leverages contrastive learning to learn meaningful representations by comparing augmented versions of the same image.","title":"Day 29: Exploring SimCLR for Self-Supervised Learning - Part 1","type":"challenge"},{"content":"Today, I fine-tuned the BERT model on the IMDb dataset for a custom NLP task: sentiment analysis. Fine-tuning allows the pre-trained model to adapt to specific tasks and datasets, resulting in better performance compared to training from scratch.\nProblem Statement # The task was to classify movie reviews from the IMDb dataset as:\nPositive Sentiment: Label 1 Negative Sentiment: Label 0 The dataset consists of 50,000 movie reviews equally split into training and testing sets.\nFine-Tuning BERT # Code: # # Problem: Fine-tune the BERT model on a custom NLP task from transformers import BertTokenizer, TFBertForSequenceClassification from datasets import load_dataset from tensorflow.keras.optimizers import Adam from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau # Load the data # This dataset contains 50,000 movie reviews, # split equally into training and testing sets, # with labels indicating whether the review is positive (1) or negative (0). dataset = load_dataset(\u0026#39;imdb\u0026#39;) # Load the Bert Tokenizer tokenizer = BertTokenizer.from_pretrained(\u0026#39;bert-base-uncased\u0026#39;) def tokenize_func(movie): return tokenizer(movie[\u0026#39;text\u0026#39;], padding=\u0026#39;max_length\u0026#39;, truncation=True, max_length=128) # Tokenize the dataset tokenized_dataset = dataset.map(tokenize_func, batched=True) # Prepare the Dataset # Convert the tokenized dataset into a TensorFlow-friendly format. train_dataset = tokenized_dataset[\u0026#39;train\u0026#39;].to_tf_dataset( columns=[\u0026#39;input_ids\u0026#39;, \u0026#39;attention_mask\u0026#39;], label_cols=\u0026#39;label\u0026#39;, shuffle=True, batch_size=16 ) test_dataset = tokenized_dataset[\u0026#39;test\u0026#39;].to_tf_dataset( columns=[\u0026#39;input_ids\u0026#39;, \u0026#39;attention_mask\u0026#39;], label_cols=\u0026#39;label\u0026#39;, shuffle=False, batch_size=16 ) # Build Bert model for classification model = TFBertForSequenceClassification.from_pretrained(\u0026#39;bert-base-uncased\u0026#39;, num_labels=2) model.compile( optimizer=Adam(learning_rate=0.00005), loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;] ) # Add callbacks for better fine-tunning checkpoint_callback = ModelCheckpoint( filepath=\u0026#39;bert_finetuned_best_model.h5\u0026#39;, save_best_only=True, save_weights_only=True, monitor=\u0026#39;val_loss\u0026#39;, mode=\u0026#39;min\u0026#39;, verbose=1 ) early_stopping_callback = EarlyStopping( monitor=\u0026#39;val_loss\u0026#39;, patience=3, restore_best_weights=True, verbose=1 ) reduce_lr_callback = ReduceLROnPlateau( monitor=\u0026#39;val_loss\u0026#39;, patience=2, factor=0.5, min_lr=0.000006, verbose=1 ) # Train the model model.fit( train_dataset, validation_data=test_dataset, epochs=10, callbacks=[checkpoint_callback, early_stopping_callback, reduce_lr_callback] ) # Load the best model for evaluation. model.load_weights(\u0026#39;bert_finetuned_best_model.h5\u0026#39;) # Evaluate the Model loss, accuracy = model.evaluate(test_dataset) print(f\u0026#34;Loss is {loss} and accuracy is: {accuracy}\u0026#34;) Step 1: Dataset Preparation # The IMDb dataset was loaded using the datasets library. Reviews were tokenized using the bert-base-uncased tokenizer: Padding: Ensures input sequences are of equal length. Truncation: Trims longer reviews to a maximum length of 128 tokens. Max Length: Limits the tokenized sequences to 128 tokens. The tokenized dataset was converted into a TensorFlow-friendly format using the to_tf_dataset method.\nStep 2: BERT Model Configuration # The TFBertForSequenceClassification model was used:\nPre-trained Weights: bert-base-uncased. Classification Head: A fully connected layer with 2 output neurons for binary classification. Compilation Details:\nOptimizer: Adam with a learning rate of 0.00005. Loss Function: sparse_categorical_crossentropy for binary classification. Metric: Accuracy. Step 3: Callbacks for Fine-Tuning # Several callbacks were added to improve fine-tuning:\nModel Checkpoint: Saves the best model weights based on validation loss. Early Stopping: Stops training if the validation loss does not improve for 3 consecutive epochs. Restores the best weights at the end of training. Reduce Learning Rate on Plateau: Reduces the learning rate by a factor of 0.5 if validation loss stagnates for 2 epochs. Prevents the model from getting stuck in a plateau. Step 4: Training the Model # The model was trained for 10 epochs with:\nBatch Size: 16 for both training and validation datasets. Validation Data: Testing dataset was used for validation during training. Callbacks: The three callbacks ensured efficient and effective training. Step 5: Evaluation # After training, the model\u0026rsquo;s best weights (saved during training) were loaded for evaluation. The model was tested on the test dataset to calculate:\nLoss: Measures the error in predictions. Accuracy: Measures the percentage of correctly classified reviews. Results # The fine-tuned model achieved the following results:\nLoss: Approximately 0.27 Accuracy: 0.93 (93%) Video # ","date":"28 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_28/","section":"Challenges","summary":"Today, I fine-tuned the BERT model on the IMDb dataset for a custom NLP task: sentiment analysis. Fine-tuning allows the pre-trained model to adapt to specific tasks and datasets, resulting in better performance compared to training from scratch.","title":"Day 28: Fine-Tuning the BERT Model for Sentiment Analysis","type":"challenge"},{"content":"Today, I implemented a BERT-based transformer model to classify movie reviews as either positive or negative using the IMDb dataset. This was my first dive into transformers for text classification, and it was an exciting exploration into natural language processing (NLP) using state-of-the-art models.\nProblem Statement # The task was to classify movie reviews from the IMDb dataset as either:\nPositive: Label 1 Negative: Label 0 The dataset consists of 50,000 movie reviews, split equally into training and testing sets.\nApproach # Code: # # Problem: Build a simple transformer-based model (BERT) for text classification (IMDb Dataset) from transformers import BertTokenizer, TFBertForSequenceClassification from datasets import load_dataset from tensorflow.keras.optimizers import Adam # Load the data # This dataset contains 50,000 movie reviews, # split equally into training and testing sets, # with labels indicating whether the review is positive (1) or negative (0). dataset = load_dataset(\u0026#39;imdb\u0026#39;) # Load the Bert Tokenizer tokenizer = BertTokenizer.from_pretrained(\u0026#39;bert-base-uncased\u0026#39;) def tokenize_func(movie): return tokenizer(movie[\u0026#39;text\u0026#39;], padding=\u0026#39;max_length\u0026#39;, truncation=True, max_length=128) # Tokenize the dataset tokenized_dataset = dataset.map(tokenize_func, batched=True) # Prepare the Dataset # Convert the tokenized dataset into a TensorFlow-friendly format. train_dataset = tokenized_dataset[\u0026#39;train\u0026#39;].to_tf_dataset( columns=[\u0026#39;input_ids\u0026#39;, \u0026#39;attention_mask\u0026#39;], label_cols=\u0026#39;label\u0026#39;, shuffle=True, batch_size=16 ) test_dataset = tokenized_dataset[\u0026#39;test\u0026#39;].to_tf_dataset( columns=[\u0026#39;input_ids\u0026#39;, \u0026#39;attention_mask\u0026#39;], label_cols=\u0026#39;label\u0026#39;, shuffle=False, batch_size=16 ) # Build Bert model for classification model = TFBertForSequenceClassification.from_pretrained(\u0026#39;bert-base-uncased\u0026#39;, num_labels=2) model.compile( optimizer=Adam(learning_rate=0.00005), loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;] ) # Train the model model.fit( train_dataset, validation_data=test_dataset, epochs=3 ) # Evaluate the Model loss, accuracy = model.evaluate(test_dataset) print(f\u0026#34;Loss is {loss} and accuracy is: {accuracy}\u0026#34;) Step 1: Dataset Loading # The IMDb dataset was loaded using the datasets library:\nTraining Set: 25,000 movie reviews. Testing Set: 25,000 movie reviews. The reviews were then tokenized to make them compatible with the BERT model.\nStep 2: Tokenization with BERT Tokenizer # I used the pre-trained bert-base-uncased tokenizer from the Hugging Face Transformers library:\nPadding: Ensures all input sequences are of equal length. Truncation: Trims longer reviews to a maximum length of 128 tokens. Max Length: Limits the tokenized sequences to 128 tokens for efficient training. The tokenized dataset was then converted into a TensorFlow-friendly format using the to_tf_dataset method, which supports:\nInput Columns: input_ids and attention_mask. Labels: Positive or negative sentiment (label column). Step 3: Model Architecture # The model used was TFBertForSequenceClassification:\nPre-trained Weights: bert-base-uncased, which has already been trained on a large corpus of English text. Classification Head: A simple fully connected layer with 2 output neurons (for binary classification). The model was compiled with:\nOptimizer: Adam with a learning rate of 0.00005. Loss Function: sparse_categorical_crossentropy for multi-class classification. Metric: Accuracy. Step 4: Training # The model was trained on the tokenized training set for 3 epochs with the following:\nBatch Size: 16 for both training and testing datasets. Validation Data: Testing dataset was used for validation during training. Step 5: Evaluation # After training, the model was evaluated on the test dataset to calculate:\nLoss: Indicates the error in predictions. Accuracy: Measures how many reviews were correctly classified. Video # ","date":"27 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_27/","section":"Challenges","summary":"Today, I implemented a BERT-based transformer model to classify movie reviews as either positive or negative using the IMDb dataset. This was my first dive into transformers for text classification, and it was an exciting exploration into natural language processing (NLP) using state-of-the-art models.","title":"Day 27: Building a Transformer-Based Model (BERT) for Text Classification on IMDb Dataset","type":"challenge"},{"content":"Today, I trained the CycleGAN model that I implemented on Day 25. The training involved optimizing both the generators and discriminators, ensuring that the generated images preserve the style and features of the target domain. This marks an exciting step in the process of unpaired image-to-image translation.\nCode: # # Set up the directories to save the generted images if not os.path.exists(\u0026#39;generated_images\u0026#39;): os.makedirs(\u0026#34;generated_images\u0026#34;) # Define the Hyper-parameters EPOCHS = 100 BATCH_SIZE = 1 SAVE_INTERVAL = 10 # Label for real and fake images for training discriminator REAL_LABEL = np.ones((BATCH_SIZE, 16, 16, 1)) # Will be of shape 16x16 with 1 channel FAKE_LABEL = np.zeros((BATCH_SIZE, 16, 16, 1)) for epoch in range(EPOCHS): for real_a, real_b in Dataset.zip((train_horses, train_zebras)).take(100): # Generate fake images using the generator fake_a = generator_g.predict(real_a) fake_b = generator_f.predict(real_b) # Train discriminator with real and fake images # Train Discriminator A d_a_real_loss = discriminator_a.train_on_batch(real_a, REAL_LABEL) d_a_fake_loss = discriminator_a.train_on_batch(fake_a, FAKE_LABEL) d_a_loss = 0.5 * np.add(d_a_real_loss, d_a_fake_loss) # Train Discriminator B d_b_real_loss = discriminator_b.train_on_batch(real_b, REAL_LABEL) d_b_fake_loss = discriminator_b.train_on_batch(fake_b, FAKE_LABEL) d_b_loss = 0.5 * np.add(d_b_real_loss, d_b_fake_loss) # Train generator to fool discriminator and maintain cycle consistency g_loss = combined_model.train_on_batch([real_a, real_b], [REAL_LABEL, REAL_LABEL, real_a, real_b, real_a, real_b]) # Print the progress print(f\u0026#34;Epoch: {epoch + 1} / {EPOCHS}\u0026#34;) print(f\u0026#34;D_A_Loss: {d_a_loss[0]:.4f}, D_B_LOSS: {d_b_loss[0]:.4f}\u0026#34;) print(f\u0026#34;G_loss: {g_loss}\u0026#34;) # Save generated images at regular interval. if (epoch + 1) % SAVE_INTERVAL == 0: fake_a = generator_g.predict(real_a) fake_b = generator_f.predict(real_b) # Visualize the generated images plt.figure(figsize=(10, 6)) plt.subplot(2, 2, 1) plt.title(\u0026#34;Original Horse\u0026#34;) plt.imshow((real_a[0] + 1) / 2) # Rescale to [0,1] plt.axis(\u0026#39;off\u0026#39;) plt.subplot(2, 2, 2) plt.title(\u0026#34;Generated Zebra\u0026#34;) plt.imshow((fake_b[0] + 1) / 2) # Rescale to [0,1] plt.axis(\u0026#39;off\u0026#39;) plt.subplot(2, 2, 3) plt.title(\u0026#34;Original Zebra\u0026#34;) plt.imshow((real_b[0] + 1) / 2) # Rescale to [0,1] plt.axis(\u0026#39;off\u0026#39;) plt.subplot(2, 2, 4) plt.title(\u0026#34;Generated Horse\u0026#34;) plt.imshow((fake_a[0] + 1) / 2) # Rescale to [0,1] plt.axis(\u0026#39;off\u0026#39;) plt.savefig(f\u0026#34;generated_images/epochs_{epoch + 1}.png\u0026#34;) plt.show() Step 1: Setting Up the Training Loop # Epochs: The model was trained for 100 epochs. Batch Size: A batch size of 1 was used, as CycleGAN models typically use smaller batches for better style preservation. Saving Interval: Generated images were saved every 10 epochs for visual evaluation. Step 2: Loss Functions # Discriminator Loss:\nFor each discriminator (D_A for horses and D_B for zebras), real and fake images were passed through the network. The losses for real and fake images were combined to update the discriminator. Generator Loss:\nIncludes adversarial loss to fool the discriminator. Cycle consistency loss to ensure that translating an image to another domain and back reconstructs the original image. Identity loss to preserve the original features when mapping an image to the same domain. Step 3: Training the Discriminator # Real images were labeled as 1 (real), and generated images were labeled as 0 (fake). The discriminator loss was computed by averaging the real and fake image losses. Step 4: Training the Generator # The combined CycleGAN model was trained with the generator and discriminator losses, along with cycle consistency and identity losses. Sample Outputs # During training, I saved the generated images at regular intervals to monitor progress.\nExample Outputs at Epoch 100 Example Outputs at Epoch 100 Video # ","date":"26 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_26/","section":"Challenges","summary":"Today, I trained the CycleGAN model that I implemented on Day 25. The training involved optimizing both the generators and discriminators, ensuring that the generated images preserve the style and features of the target domain. This marks an exciting step in the process of unpaired image-to-image translation.","title":"Day 26: Training the CycleGAN for Style Transfer - Part 2: Horse to Zebra Conversion","type":"challenge"},{"content":"Today, I began implementing CycleGAN, a type of Generative Adversarial Network (GAN) designed for unpaired image-to-image translation. This project focuses on style transfer, specifically converting images of horses into zebras and vice versa. CycleGAN enables style transformation without needing paired datasets, which makes it versatile and powerful for many real-world applications.\nIn this first part, I focused on:\nLoading and preprocessing the dataset. Building the generator and discriminator networks. Constructing the combined CycleGAN model. What is a CycleGAN? # CycleGAN is a GAN-based architecture for unpaired image-to-image translation. Unlike traditional GANs, CycleGAN uses cycle consistency loss, which ensures that translating an image to another domain and back results in the original image.\nKey Components: # Generators:\nG: Transforms images from Domain A (horses) to Domain B (zebras). F: Transforms images from Domain B (zebras) to Domain A (horses). Discriminators:\nD_A: Distinguishes real horses from fake horses generated by F. D_B: Distinguishes real zebras from fake zebras generated by G. Cycle Consistency Loss:\nEnsures that when an image is transformed from one domain to another and then back, it closely resembles the original. Code # # Problem: Implement CycleGAN for style transfer (e.g., horse to zebra conversion) from tensorflow.keras.layers import Input, Conv2D, Conv2DTranspose, LeakyReLU, ReLU, BatchNormalization, Concatenate from tensorflow.keras.models import Model import numpy as np import matplotlib.pyplot as plt import os from glob import glob from tensorflow.keras.optimizers import Adam from tensorflow.keras.preprocessing.image import load_img, img_to_array from tensorflow.data import Dataset # Load the dataset HORSE_DIR = \u0026#39;dataset/horse2zebra/trainA/\u0026#39; ZEBRA_DIR = \u0026#39;dataset/horse2zebra/trainB/\u0026#39; # Helper function to load images from directories def load_images_from_directory(directory, size=(128, 128)): images = [] for filepath in glob(os.path.join(directory, \u0026#39;*.jpg\u0026#39;)): image = load_img(filepath, target_size=size) image = img_to_array(image) images.append(image) return np.array(images) # Load horses and zebra images horse_images = load_images_from_directory(HORSE_DIR) zebra_images = load_images_from_directory(ZEBRA_DIR) # Normalize it between [-1, 1] horse_images = (horse_images - 127.5) / 127.5 zebra_images = (zebra_images - 127.5) / 127.5 # Convert to tensorflow dataset and batch them train_horses = Dataset.from_tensor_slices(horse_images).batch(1) train_zebras = Dataset.from_tensor_slices(zebra_images).batch(1) # Build the Generator Model def build_generator(): inputs = Input(shape=(128, 128, 3)) # Encoder: Downsampling layers x = Conv2D(64, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(inputs) x = LeakyReLU(alpha=0.2)(x) x = Conv2D(128, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(x) x = BatchNormalization()(x) x = LeakyReLU(alpha=0.2)(x) x = Conv2D(256, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(x) x = BatchNormalization()(x) x = LeakyReLU(alpha=0.2)(x) # Decoder: Upsampling layers x = Conv2DTranspose(128, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(x) x = BatchNormalization()(x) x = ReLU()(x) x = Conv2DTranspose(64, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(x) x = BatchNormalization()(x) x = ReLU()(x) x = Conv2DTranspose(3, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;, activation=\u0026#39;tanh\u0026#39;)(x) return Model(inputs, x) # Build generator for both transformations generator_g = build_generator() # Horse to zebra generator_f = build_generator() # Zebra to horse generator_g.summary() generator_f.summary() # Build the discriminator Model # Define the discriminator model def build_discriminator(): inputs = Input(shape=(128, 128, 3)) x = Conv2D(64, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(inputs) x = LeakyReLU(alpha=0.2)(x) x = Conv2D(128, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(x) x = BatchNormalization()(x) x = LeakyReLU(alpha=0.2)(x) x = Conv2D(256, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(x) x = BatchNormalization()(x) x = LeakyReLU(alpha=0.2)(x) x = Conv2D(1, kernel_size=4, padding=\u0026#39;same\u0026#39;)(x) return Model(inputs, x) # Build the dicriminator for both domains discriminator_a = build_discriminator() # For Domain A (Horses) discriminator_b = build_discriminator() # For Domain B (Zebras) discriminator_a.compile( optimizer=Adam(learning_rate=0.0002, beta_1=0.5), loss=\u0026#39;mse\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;] ) discriminator_b.compile( optimizer=Adam(learning_rate=0.0002, beta_1=0.5), loss=\u0026#39;mse\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;] ) discriminator_a.summary() discriminator_b.summary() # Build Cycle GAN Model # Define the combined cycle GAN model def build_combined(generator_g, generator_f, discriminator_a, discriminator_b): discriminator_a.trainable = False discriminator_b.trainable = False # Real input images for both the domain input_a = Input(shape=(128, 128, 3)) # Horses input_b = Input(shape=(128, 128, 3)) # Zebras # Forward cycle: A -\u0026gt; B -\u0026gt; A fake_b = generator_g(input_a) cycle_a = generator_f(fake_b) # Backward cycle: B -\u0026gt; A -\u0026gt; B fake_a = generator_f(input_b) cycle_b = generator_g(fake_a) # Identifying mapping preserving original features same_a = generator_f(input_a) same_b = generator_g(input_b) # Discriminators for the generated images valid_a = discriminator_a(fake_a) valid_b = discriminator_b(fake_b) # Define the combined model model = Model( inputs=[input_a, input_b], outputs=[ valid_a, valid_b, cycle_a, cycle_b, same_a, same_b ] ) model.compile( optimizer=Adam(learning_rate=0.0002, beta_1=0.5), loss=[\u0026#39;mse\u0026#39;, \u0026#39;mse\u0026#39;, \u0026#39;mse\u0026#39;, \u0026#39;mse\u0026#39;, \u0026#39;mse\u0026#39;, \u0026#39;mse\u0026#39;], loss_weights=[1, 1, 10, 10, 5, 5] ) return model combined_model = build_combined(generator_g, generator_f, discriminator_a, discriminator_b) combined_model.summary() Step 1: Dataset Preparation # I used the horse2zebra dataset from CycleGAN\u0026rsquo;s original implementation.\nLoading Images:\nImages were loaded using the glob module to iterate over files in the dataset folders. Resized to 128x128 for faster computation. Normalization:\nPixel values were normalized to the range [-1, 1] to match the output of the generator\u0026rsquo;s tanh activation function. Batching:\nThe preprocessed images were converted into TensorFlow datasets and batched for training. Step 2: Building the Generator # The generator architecture consists of:\nEncoder: Downsampling layers using Conv2D and LeakyReLU. Decoder: Upsampling layers using Conv2DTranspose and ReLU. Final layer with a tanh activation to generate images in the range [-1, 1]. Two Generators:\nG: Converts horses to zebras. F: Converts zebras to horses. Step 3: Building the Discriminator # The discriminator architecture:\nUses Conv2D layers for feature extraction and downsampling. LeakyReLU activation is applied after each layer. Outputs a single value indicating whether the input image is real or fake. Two Discriminators:\nD_A: Classifies images in Domain A (horses). D_B: Classifies images in Domain B (zebras). Each discriminator is trained to minimize the mean squared error (mse) loss.\nStep 4: Combining the Models # The combined CycleGAN model includes:\nForward Cycle: A -\u0026gt; B -\u0026gt; A: Translates a horse to a zebra and back to a horse. Backward Cycle: B -\u0026gt; A -\u0026gt; B: Translates a zebra to a horse and back to a zebra. Identity Mapping: G(A) ≈ A: Ensures that translating an image from one domain to itself preserves its features. Discriminator Feedback: D_A and D_B provide feedback to the generators. Loss Functions:\nAdversarial Loss: Encourages generators to produce realistic images. Cycle Consistency Loss: Penalizes discrepancies between input and reconstructed images. Identity Loss: Preserves color and style during translation. Challenges Faced # Balancing Loss Terms: Combining multiple loss terms with different weights required careful tuning. Resource Requirements: The model is computationally intensive due to its two generators and two discriminators. Image Quality: Initial results were blurry, likely due to limited training data and early-stage model adjustments. Next Steps # In the next part of this project, I’ll focus on:\nTraining the CycleGAN model: Implementing the training loop for the generators and discriminators. Monitoring loss values and generated images to ensure convergence. Evaluating Results: Visualizing transformed images during training. Comparing generated outputs to assess cycle consistency. This is an exciting exploration into unpaired style transfer with CycleGAN. Stay tuned for Part 2, where I\u0026rsquo;ll dive into training the model and visualizing the results!\nVideo # ","date":"25 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_25/","section":"Challenges","summary":"Today, I began implementing CycleGAN, a type of Generative Adversarial Network (GAN) designed for unpaired image-to-image translation. This project focuses on style transfer, specifically converting images of horses into zebras and vice versa. CycleGAN enables style transformation without needing paired datasets, which makes it versatile and powerful for many real-world applications.","title":"Day 25: Exploring CycleGAN for Style Transfer - Part 1: Model Architecture and Setup","type":"challenge"},{"content":"Today, I dived into Conditional GANs (CGANs), an exciting variation of GANs that allows for generating specific types of images conditioned on labels. Using the Fashion MNIST dataset, I implemented a CGAN to generate images of specific clothing items like shirts, shoes, and bags.\nWhat is a Conditional GAN? # A Conditional GAN (CGAN) is an extension of GANs where the generation of data is conditioned on some additional information, such as labels or attributes. In this project:\nThe generator takes both noise (random input) and a label (e.g., \u0026ldquo;sneaker\u0026rdquo;). The discriminator evaluates whether an image-label pair is real or fake. This setup allows the CGAN to generate specific types of images based on the provided label.\nCode: # # Problem: Conditional GAN (CGAN) for Generating Specific Images from Fashion MNIST import numpy as np import tensorflow as tf from tensorflow.keras.layers import Dense, Flatten, Reshape, LeakyReLU, BatchNormalization, Conv2DTranspose, Conv2D, \\ Input, Concatenate from tensorflow.keras.models import Sequential, Model import matplotlib.pyplot as plt from tensorflow.keras.datasets import fashion_mnist from tensorflow.keras.optimizers import Adam from tensorflow.keras.utils import to_categorical # Load the Fashion MNIST dataset (X_train, y_train), (_, _) = fashion_mnist.load_data() # Normalize the images to the range [-1, 1] to fit the tanh activation function in the generator X_train = (X_train - 127.5) / 127.5 X_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype(\u0026#39;float32\u0026#39;) # One-hot encode the labels for conditioning num_classes = 10 y_train = to_categorical(y_train, num_classes) # Function to build the generator def build_generator(): # Inputs for the generator noise_input = Input(shape=(100,)) label_input = Input(shape=(num_classes,)) # Concatenate noise and label to create the input for the generator model_input = Concatenate()([noise_input, label_input]) x = Dense(7 * 7 * 256, activation=\u0026#39;relu\u0026#39;)(model_input) x = Reshape((7, 7, 256))(x) x = BatchNormalization(momentum=0.8)(x) # Upsample to 14x14 x = Conv2DTranspose(128, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(x) x = LeakyReLU(alpha=0.2)(x) x = BatchNormalization(momentum=0.8)(x) # Upsample to 28x28 x = Conv2DTranspose(64, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)(x) x = LeakyReLU(alpha=0.2)(x) x = BatchNormalization(momentum=0.8)(x) # Final layer to generate an image with 28x28 dimensions and 1 channel img_output = Conv2D(1, kernel_size=7, activation=\u0026#39;tanh\u0026#39;, padding=\u0026#39;same\u0026#39;)(x) return Model([noise_input, label_input], img_output) # Build the generator model generator = build_generator() generator.summary() # Function to build the discriminator def build_discriminator(): # Inputs for the image and the label img_input = Input(shape=(28, 28, 1)) label_input = Input(shape=(num_classes,)) # Embed the label and reshape to match the image shape label_embedding = Dense(28 * 28)(label_input) label_embedding = Reshape((28, 28, 1))(label_embedding) # Concatenate the image and label embedding combined_input = Concatenate()([img_input, label_embedding]) # Flatten the combined input and pass through dense layers x = Flatten()(combined_input) x = Dense(512)(x) x = LeakyReLU(alpha=0.2)(x) x = Dense(256)(x) x = LeakyReLU(alpha=0.2)(x) # Final output layer to classify real (1) or fake (0) validity_output = Dense(1, activation=\u0026#39;sigmoid\u0026#39;)(x) # Create the model that takes the image and label as input return Model([img_input, label_input], validity_output) # Build and compile the discriminator model discriminator = build_discriminator() discriminator.compile( optimizer=Adam(learning_rate=0.0002), loss=\u0026#39;binary_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;] ) discriminator.summary() # Build the combined CGAN model # Freeze the discriminator\u0026#39;s layers when training the combined CGAN model discriminator.trainable = False # Inputs for noise and label noise_input = Input(shape=(100,)) label_input = Input(shape=(num_classes,)) # Generate an image from the noise and label input img = generator([noise_input, label_input]) # Use the discriminator to classify the generated image with the label validity = discriminator([img, label_input]) # Define the combined CGAN model cgan = Model([noise_input, label_input], validity) cgan.compile(optimizer=Adam(learning_rate=0.0002), loss=\u0026#39;binary_crossentropy\u0026#39;) cgan.summary() # Training the CGAN # Training Parameters epochs = 10000 batch_size = 32 save_interval = 1000 # Labels for real and fake images real = np.ones((batch_size, 1)) * 0.9 # Smoothed label for real images fake = np.zeros((batch_size, 1)) + 0.1 # Noisy label for fake images for epoch in range(epochs): # Train the discriminator with real images idx = np.random.randint(0, X_train.shape[0], batch_size) real_imgs = X_train[idx] real_labels = y_train[idx] # Generate fake images noise = np.random.normal(0, 1, (batch_size, 100)) fake_labels = np.eye(num_classes)[np.random.choice(num_classes, batch_size)] gen_imgs = generator.predict([noise, fake_labels]) # Train the discriminator on real and fake images d_loss_real = discriminator.train_on_batch([real_imgs, real_labels], real) d_loss_fake = discriminator.train_on_batch([gen_imgs, fake_labels], fake) d_loss = 0.5 * np.add(d_loss_real, d_loss_fake) # Train the generator via the combined CGAN model noise = np.random.normal(0, 1, (batch_size, 100)) sampled_labels = np.eye(num_classes)[np.random.choice(num_classes, batch_size)] g_loss = cgan.train_on_batch([noise, sampled_labels], real) # Display training progress and save images at intervals if epoch % save_interval == 0: print(f\u0026#34;{epoch} [D loss: {d_loss[0]}, acc.: {100 * d_loss[1]}%] [G loss: {g_loss}]\u0026#34;) # Save generated images to visualize training progress generated_imgs = generator.predict([noise, sampled_labels]) generated_imgs = 0.5 * generated_imgs + 0.5 # Rescale from [-1, 1] to [0, 1] plt.figure(figsize=(5, 5)) for i in range(4): plt.subplot(2, 2, i + 1) plt.imshow(generated_imgs[i, :, :, 0], cmap=\u0026#39;gray\u0026#39;) plt.axis(\u0026#39;off\u0026#39;) plt.show() Steps Implemented # Step 1: Dataset Preparation # Dataset: I used the Fashion MNIST dataset, which contains grayscale 28x28 images of clothing items across 10 classes. Normalization: Images were normalized to the range [-1, 1] to match the output of the generator\u0026rsquo;s tanh activation. One-Hot Encoding: Labels were converted to a one-hot encoding format to condition both the generator and discriminator. Step 2: Building the Generator # The generator takes:\nNoise: A random 100-dimensional vector. Label: A one-hot encoded label vector. The inputs are concatenated and passed through:\nDense and reshaping layers to form a low-resolution image. Transposed convolutions to upsample the image to 28x28. Batch normalization to stabilize training and speed up convergence. A final layer with tanh activation to output an image. Objective: Generate a realistic image conditioned on the label.\nStep 3: Building the Discriminator # The discriminator takes:\nImage: A 28x28 grayscale image. Label: A one-hot encoded label embedded and reshaped to match the image dimensions. The inputs are concatenated and passed through:\nDense layers with LeakyReLU activation for feature extraction. A final dense layer with sigmoid activation to classify the input as real or fake. Objective: Distinguish between real and fake image-label pairs.\nStep 4: Building the CGAN Model # The CGAN combines the generator and discriminator:\nThe generator outputs a fake image given noise and a label. The discriminator evaluates the generated image-label pair. The generator is trained to fool the discriminator, encouraging it to produce realistic images. The discriminator’s layers are frozen while training the combined CGAN model.\nStep 5: Training the CGAN # Discriminator Training: Trained on real and fake images with smoothed labels to improve stability. Generator Training: Trained using the combined CGAN model to maximize the discriminator\u0026rsquo;s classification error on fake images. Step 6: Visualizing Results # During training:\nImages were generated at regular intervals to monitor progress. Noise and labels were sampled to visualize specific clothing items like \u0026ldquo;sneakers\u0026rdquo; or \u0026ldquo;shirts.\u0026rdquo; Video # ","date":"24 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_24/","section":"Challenges","summary":"Today, I dived into Conditional GANs (CGANs), an exciting variation of GANs that allows for generating specific types of images conditioned on labels. Using the Fashion MNIST dataset, I implemented a CGAN to generate images of specific clothing items like shirts, shoes, and bags.","title":"Day 24: Exploring Conditional GANs (CGANs) with Fashion MNIST","type":"challenge"},{"content":"Welcome to Day 23 of our deep learning challenge! Today, we will discuss the improvements made to the Generative Adversarial Network (GAN) model to generate clearer images for the Fashion MNIST dataset. We\u0026rsquo;ll explore the changes made to the generator, discriminator, and the overall training process to help enhance the output quality. Let\u0026rsquo;s dive into the improvements and understand why they were made.\nStep 1: Importing Libraries and Loading Data # We start by importing necessary libraries and loading the Fashion MNIST dataset.\nimport numpy as np import tensorflow as tf from tensorflow.keras.layers import Dense, Flatten, Reshape, LeakyReLU, BatchNormalization, Conv2DTranspose, Conv2D from tensorflow.keras.models import Sequential import matplotlib.pyplot as plt from tensorflow.keras.datasets import fashion_mnist import pandas as pd import os from tensorflow.keras.optimizers import Adam # Load the data (X_train, _), (_, _) = fashion_mnist.load_data() # Normalize between -1 and 1 as it helps tanh activation function. X_train = (X_train - 127.5) / 127.5 X_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype(\u0026#39;float32\u0026#39;) Normalization: The data is normalized between -1 and 1 to match the output range of the tanh activation used in the generator. This helps the model learn more effectively. Reshaping: The data is reshaped to have a single channel (28, 28, 1) to fit the expected input shape of the generator and discriminator. Step 2: Generator Improvements # Original Generator # The original generator used fully connected layers to upscale the latent space into a 28x28 image. While this approach can work, it struggles with spatial resolution and generating detailed images.\nImproved Generator # The improved generator uses transposed convolution layers (Conv2DTranspose) to better handle upsampling and generate clearer images.\ndef build_improved_generator(): model = Sequential() model.add(Dense(7 * 7 * 256, input_dim=100)) model.add(LeakyReLU(alpha=0.2)) model.add(Reshape((7, 7, 256))) # Upsampling to 14x14 model.add(Conv2DTranspose(128, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)) model.add(LeakyReLU(alpha=0.2)) model.add(BatchNormalization(momentum=0.8)) # Upsampling to 28x28 model.add(Conv2DTranspose(64, kernel_size=4, strides=2, padding=\u0026#39;same\u0026#39;)) model.add(LeakyReLU(alpha=0.2)) model.add(BatchNormalization(momentum=0.8)) # Final layer to generate images model.add(Conv2D(1, kernel_size=7, activation=\u0026#39;tanh\u0026#39;, padding=\u0026#39;same\u0026#39;)) return model Changes Explained: # Dense Layer with Reshape: The generator starts with a Dense layer that outputs a shape of (7, 7, 256) followed by reshaping, which helps form a low-resolution base to build upon. Transposed Convolutions (Conv2DTranspose): Instead of fully connected layers, transposed convolutions are used to gradually upscale the image to 28x28. This helps retain spatial hierarchies and generates clearer images. LeakyReLU Activation: The LeakyReLU activation is used to avoid dead neurons and enhance the flow of gradients. It uses a small slope for negative values (alpha=0.2), allowing some negative gradient flow. Batch Normalization: Helps stabilize training and enables the model to converge faster by normalizing activations. Step 3: Discriminator Improvements # The original discriminator consisted of fully connected layers which were not ideal for extracting spatial features from images. Therefore, we kept the fully connected layers but considered replacing them with convolutional layers to improve the performance. However, in this version, we still retained the basic structure with some changes in training.\ndef build_discriminator(): model = Sequential() model.add(Flatten(input_shape=(28, 28, 1))) # Add layers to process flatten image model.add(Dense(512)) model.add(LeakyReLU(alpha=0.2)) model.add(Dense(256)) model.add(LeakyReLU(alpha=0.2)) # Final output layer to classify real(1) or fake(0) model.add(Dense(1, activation=\u0026#39;sigmoid\u0026#39;)) return model Changes Explained: # Flatten Layer: The Flatten layer is used to convert the image into a 1D vector. LeakyReLU Activation: This activation function is used after each Dense layer, making the network more resilient against the vanishing gradient problem. Compilation and Learning Rate Changes # The discriminator uses a learning rate of 0.0002 for the Adam optimizer, and the GAN uses a smaller rate of 0.0001. These lower rates help make the training more stable. Step 4: Building the GAN # To train both the generator and discriminator as a combined model, we freeze the discriminator\u0026rsquo;s weights and define the GAN model.\ndef build_gan(generator, discriminator): model = Sequential() model.add(generator) model.add(discriminator) return model discriminator.trainable = False gan = build_gan(generator, discriminator) gan.compile(optimizer=Adam(learning_rate=0.0001), loss=\u0026#39;binary_crossentropy\u0026#39;) Changes Explained: # Discriminator.trainable = False: This line freezes the discriminator while training the GAN. We do not want discriminator weights to update when we are training the generator, as the GAN\u0026rsquo;s objective is to trick the discriminator. Step 5: Training the GAN # We made multiple improvements to the training process to help stabilize and enhance the quality of generated images.\n# Training Parameters epochs = 30000 batch_size = 16 save_intervals = 1000 # Label for real and fake images real = np.ones((batch_size, 1)) * 0.9 fake = np.zeros((batch_size, 1)) + 0.1 for epoch in range(epochs): # Train the discriminator random_idx = np.random.randint(0, X_train.shape[0], batch_size) real_images = X_train[random_idx] noise = np.random.normal(0, 1, (batch_size, 100)) generated_images = generator.predict(noise) # Random flips to add noise to discriminator if np.random.rand() \u0026lt; 0.1: real, fake = fake, real d_loss_real = discriminator.train_on_batch(real_images, real) d_loss_fake = discriminator.train_on_batch(generated_images, fake) d_loss = 0.5 * np.add(d_loss_real, d_loss_fake) # Train the Generator # Train the generator twice to give it more opportunity. for _ in range(2): noise = np.random.normal(0, 1, (batch_size, 100)) gan_loss = gan.train_on_batch(noise, real) if epoch % save_intervals == 0: print(f\u0026#34;{epoch} [D loss: {d_loss[0]}, acc.: {100 * d_loss[1]}%] [G loss: {gan_loss}]\u0026#34;) # Save the generated images to visualize training progress generated_images = generator.predict(noise) generated_images = 0.5 * generated_images + 0.5 # Rescale from -1 to 1 to 0 to 1 plt.figure(figsize=(5, 5)) for i in range(4): plt.subplot(2, 2, i + 1) plt.imshow(generated_images[i, :, :, 0], cmap=\u0026#39;gray\u0026#39;) plt.axis(\u0026#39;off\u0026#39;) plt.show() Training Improvements: # Label Smoothing and Noise: Labels for real and fake images are smoothed (0.9 for real, 0.1 for fake) and occasionally flipped (random flip) to add noise. This prevents the discriminator from becoming overly confident, which can destabilize GAN training.\nTrain Generator Twice: The generator is trained twice per epoch to give it more opportunities to learn and keep up with the discriminator. This helps when the discriminator tends to become too powerful.\nRandom Label Flipping: Randomly flipping real and fake labels during discriminator training further ensures that the discriminator doesn\u0026rsquo;t become too dominant, which can lead to mode collapse.\nSave Interval Visualization: We save and plot generated images at intervals (every 1000 epochs). This visualization helps track the progress of the GAN and allows us to observe improvements over time.\nSummary of Improvements # Generator Architecture: Improved by using transposed convolutions to better handle spatial upsampling, leading to sharper images. Discriminator Training: Smoothing and adding noise to labels, as well as increasing training frequency of the generator, resulted in a more stable training process. Learning Rates: Different learning rates for the discriminator and GAN helped maintain balance and stability during training. Random Label Flipping: This added robustness to the discriminator\u0026rsquo;s training and prevented overfitting. These changes significantly enhanced the performance of the GAN, helping it generate more realistic Fashion MNIST images and stabilizing the training process.\nVideo # ","date":"23 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_23/","section":"Challenges","summary":"Welcome to Day 23 of our deep learning challenge! Today, we will discuss the improvements made to the Generative Adversarial Network (GAN) model to generate clearer images for the Fashion MNIST dataset. We’ll explore the changes made to the generator, discriminator, and the overall training process to help enhance the output quality. Let’s dive into the improvements and understand why they were made.","title":"Day 23: GAN Improvements - Enhancing Performance for Fashion MNIST Generation","type":"challenge"},{"content":"Today, we explored the basics of Generative Adversarial Networks (GANs). GANs are one of the most innovative approaches in deep learning, used for generating data that closely resembles the training dataset. They have gained popularity in various fields, including art, image generation, and even data augmentation for machine learning models. In today\u0026rsquo;s session, we set up a basic GAN framework using the Fashion MNIST dataset.\nStep 1: Understanding GAN Architecture # A Generative Adversarial Network (GAN) consists of two neural networks that compete with each other:\nGenerator: This network generates synthetic data that should look like the real data. For Fashion MNIST, it generates synthetic images of clothing items. Discriminator: This network evaluates the authenticity of data, distinguishing between real and fake images. It classifies whether the input is a real image from the dataset or a fake image produced by the generator. These two models are trained in a competitive fashion, where:\nThe Generator tries to fool the Discriminator by creating realistic-looking data. The Discriminator tries to accurately identify whether the data is real or fake. This competition helps both models improve simultaneously in what is called adversarial training.\nStep 2: Import Libraries and Load Dataset # First, we import the necessary libraries and load the Fashion MNIST dataset, which is a collection of grayscale images of clothing items.\nimport numpy as np import tensorflow as tf from tensorflow.keras.layers import Dense, Flatten, Reshape, LeakyReLU, BatchNormalization from tensorflow.keras.models import Sequential import matplotlib.pyplot as plt TensorFlow/Keras: Used for building the generator and discriminator models. NumPy: Helps in handling data efficiently. Matplotlib: Used to visualize the generated images. Load the dataset:\n# Load Fashion MNIST dataset (X_train, _), (_, _) = tf.keras.datasets.fashion_mnist.load_data() # Normalize the images between -1 and 1 X_train = (X_train - 127.5) / 127.5 X_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype(\u0026#39;float32\u0026#39;) Normalization: We normalize the data to have values between -1 and 1 to match the output range of the tanh activation function used in the generator. Reshape: We reshape the images to 28x28x1 to explicitly define them as grayscale images. Step 3: Build the Generator # The Generator takes random noise and generates synthetic images that resemble real Fashion MNIST images (28x28).\n# Define the Generator def build_generator(): model = Sequential() # Dense layer to increase dimensionality from noise model.add(Dense(256, input_dim=100)) model.add(LeakyReLU(alpha=0.2)) model.add(BatchNormalization(momentum=0.8)) # Another Dense layer model.add(Dense(512)) model.add(LeakyReLU(alpha=0.2)) model.add(BatchNormalization(momentum=0.8)) # One more Dense layer model.add(Dense(1024)) model.add(LeakyReLU(alpha=0.2)) model.add(BatchNormalization(momentum=0.8)) # Final output layer, reshaping to 28x28x1 model.add(Dense(28 * 28 * 1, activation=\u0026#39;tanh\u0026#39;)) model.add(Reshape((28, 28, 1))) return model generator = build_generator() generator.summary() Dense Layers: The generator uses several dense layers to upscale a random noise vector (size 100) into an image-sized output (28x28x1). LeakyReLU Activation: LeakyReLU helps to avoid dead neurons by allowing a small gradient for negative inputs (alpha=0.2). BatchNormalization: Helps stabilize training and improves convergence speed. Output Layer: The final layer uses tanh activation, ensuring output pixel values are between -1 and 1. Step 4: Build the Discriminator # The Discriminator takes an image and outputs whether it believes the image is real or fake.\n# Define the Discriminator def build_discriminator(): model = Sequential() # Flatten the input image model.add(Flatten(input_shape=(28, 28, 1))) # Dense layer to process the flattened image model.add(Dense(512)) model.add(LeakyReLU(alpha=0.2)) # Another Dense layer model.add(Dense(256)) model.add(LeakyReLU(alpha=0.2)) # Final output layer to classify real or fake model.add(Dense(1, activation=\u0026#39;sigmoid\u0026#39;)) return model discriminator = build_discriminator() discriminator.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;binary_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) discriminator.summary() Dense Layers: Uses dense layers to analyze the input. LeakyReLU: Similar to the generator, LeakyReLU is used to allow non-zero gradients for negative values. Output Layer: Uses sigmoid activation to output a value between 0 (fake) and 1 (real). Binary Crossentropy Loss: Suitable for a binary classification problem (real vs fake). Step 5: Build and Compile the GAN # Now, let’s build and compile the GAN by combining the Generator and Discriminator.\n# Freeze the Discriminator\u0026#39;s weights during GAN training discriminator.trainable = False # Build and compile the GAN def build_gan(generator, discriminator): model = Sequential() model.add(generator) model.add(discriminator) return model gan = build_gan(generator, discriminator) gan.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;binary_crossentropy\u0026#39;) Freeze Discriminator: We freeze the discriminator\u0026rsquo;s weights while training the GAN so that the discriminator doesn\u0026rsquo;t get updated when training the generator. GAN Model: Combines the generator and discriminator so that the generator can be trained to fool the discriminator. Step 6: Training the GAN # The training loop involves iteratively training the discriminator and then the generator.\n# Training parameters epochs = 10000 batch_size = 64 save_interval = 1000 # Labels for real and fake images real = np.ones((batch_size, 1)) fake = np.zeros((batch_size, 1)) for epoch in range(epochs): # Train the Discriminator idx = np.random.randint(0, X_train.shape[0], batch_size) real_images = X_train[idx] noise = np.random.normal(0, 1, (batch_size, 100)) generated_images = generator.predict(noise) d_loss_real = discriminator.train_on_batch(real_images, real) d_loss_fake = discriminator.train_on_batch(generated_images, fake) d_loss = 0.5 * np.add(d_loss_real, d_loss_fake) # Train the Generator noise = np.random.normal(0, 1, (batch_size, 100)) g_loss = gan.train_on_batch(noise, real) # Print progress if epoch % save_interval == 0: print(f\u0026#34;{epoch} [D loss: {d_loss[0]}, acc.: {100 * d_loss[1]}%] [G loss: {g_loss}]\u0026#34;) # Save generated images to visualize training progress generated_images = generator.predict(noise) generated_images = 0.5 * generated_images + 0.5 # Rescale images from -1 to 1 to 0 to 1 plt.figure(figsize=(5, 5)) for i in range(4): plt.subplot(2, 2, i + 1) plt.imshow(generated_images[i, :, :, 0], cmap=\u0026#39;gray\u0026#39;) plt.axis(\u0026#39;off\u0026#39;) plt.show() Discriminator Training: We train the discriminator on a mix of real and fake images. Generator Training: The generator is trained via the GAN model to try and fool the discriminator. Save Interval: Every 1000 epochs, generated images are saved to monitor training progress. Summary of Performance # Initial Outputs: The initial outputs of the generator were blurry and noisy, which is expected in early epochs. Discriminator vs Generator Balance: The discriminator is often too good at the start, making it hard for the generator to improve. This leads to noisy and meaningless generated images. Some generated images:\nVideo # ","date":"22 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_22/","section":"Challenges","summary":"Today, we explored the basics of Generative Adversarial Networks (GANs). GANs are one of the most innovative approaches in deep learning, used for generating data that closely resembles the training dataset.","title":"Day 22: GAN Basics - Understanding GAN Architecture and Setting Up a GAN Framework","type":"challenge"},{"content":"Today, we continued with our autoencoder-based anomaly detection by fine-tuning the model and evaluating its performance on the Credit Card Fraud Detection Dataset. Our objectives were:\nFine-Tune the Autoencoder Model: Improve model performance by adjusting hyperparameters. Determine Reconstruction Error Threshold: Use reconstruction error to classify normal vs. fraudulent transactions. Evaluate Performance: Utilize metrics like Precision, Recall, F1-Score, and AUC to understand the model\u0026rsquo;s effectiveness. Step 1: Fine-Tuning the Autoencoder Model # In this step, we experimented with different model configurations and hyperparameters to try and reduce the reconstruction error.\nModify the Model Architecture # We experimented with the number of neurons and layers to see if a different configuration would yield better results.\n# Adjust the model complexity model = Sequential() # Encoder model.add(Dense(20, activation=\u0026#39;relu\u0026#39;, input_shape=(X_train.shape[1],))) model.add(Dense(10, activation=\u0026#39;relu\u0026#39;)) # Latent representation model.add(Dense(5, activation=\u0026#39;relu\u0026#39;)) # Smaller latent space to capture key features # Decoder model.add(Dense(10, activation=\u0026#39;relu\u0026#39;)) model.add(Dense(20, activation=\u0026#39;relu\u0026#39;)) model.add(Dense(X_train.shape[1], activation=\u0026#39;linear\u0026#39;)) # Compile with modified learning rate model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), loss=\u0026#39;mse\u0026#39;) model.summary() # Train the model history = model.fit(X_train_normal, X_train_normal, epochs=100, batch_size=128, validation_split=0.2, verbose=1) Encoder/Decoder Changes: We added more layers and neurons to increase model complexity. A smaller latent space helped in focusing on key patterns. Learning Rate: The learning rate was set to 0.001 to ensure smoother convergence. Training Epochs: Increased the number of epochs to 100 for more training time. Plot Training Loss # We plotted the training and validation loss to understand if the model was learning effectively.\nplt.plot(history.history[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss\u0026#39;) plt.plot(history.history[\u0026#39;val_loss\u0026#39;], label=\u0026#39;Validation Loss\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.legend() plt.title(\u0026#39;Training and Validation Loss for Fine-Tuned Autoencoder\u0026#39;) plt.show() A decreasing trend in both losses indicated that the model was improving in reconstruction. If there was a large gap between the two, it would indicate overfitting. Step 2: Set Threshold for Anomaly Detection # After training the autoencoder, we calculated the reconstruction error for all transactions in the test set and used this to classify transactions as either normal or anomalous.\n# Predict the reconstructed test data X_test_pred = model.predict(X_test) # Calculate reconstruction error reconstruction_errors = np.mean(np.power(X_test - X_test_pred, 2), axis=1) Reconstruction Error: Calculated using the Mean Squared Error (MSE) between X_test and X_test_pred for each data sample. Determine the Threshold # We set a threshold based on the 95th percentile of the reconstruction errors for the normal transactions in the training set.\nthreshold = np.percentile(reconstruction_errors[y_test == 0], 95) print(f\u0026#34;Threshold for anomaly detection: {threshold}\u0026#34;) A high reconstruction error indicates an anomaly since the autoencoder has difficulty reconstructing fraudulent transactions. The 95th percentile was chosen to allow for a balance between false positives and true negatives. Step 3: Classify and Evaluate # Using the reconstruction error threshold, we classified each test transaction as normal or anomalous.\ny_pred = [1 if error \u0026gt; threshold else 0 for error in reconstruction_errors] # Actual labels for evaluation print(f\u0026#34;Actual Anomalies: {sum(y_test)}, Detected: {sum(y_pred)}\u0026#34;) Threshold-based Classification: Transactions with reconstruction errors above the threshold are labeled as 1 (fraud), while those below are labeled as 0 (normal). Evaluate Model Performance # We used classification metrics to evaluate the model’s performance:\nfrom sklearn.metrics import confusion_matrix, classification_report, roc_auc_score # Confusion Matrix conf_matrix = confusion_matrix(y_test, y_pred) print(\u0026#34;Confusion Matrix:\u0026#34;) print(conf_matrix) # Classification Report print(\u0026#34;\\nClassification Report:\u0026#34;) print(classification_report(y_test, y_pred)) # AUC Score auc_score = roc_auc_score(y_test, reconstruction_errors) print(f\u0026#34;AUC Score: {auc_score:.2f}\u0026#34;) Confusion Matrix: Shows True Negatives, False Positives, False Negatives, and True Positives. Classification Report: Includes Precision, Recall, and F1-Score. Precision tells us how many flagged transactions were actually fraud. Recall tells us how many of the fraudulent transactions were detected. F1-Score is a harmonic mean of precision and recall. AUC Score: Measures the model\u0026rsquo;s ability to distinguish between positive and negative classes. A higher AUC is better. Observations from Evaluation # Threshold for anomaly detection is: 1.0146805608893366 Actual Anomalies: 98, Detected: 2932 Confusion Matrix: [[54020 2844] [ 10 88]] Classification Report: precision recall f1-score support 0 1.00 0.95 0.97 56864 1 0.03 0.90 0.06 98 accuracy 0.95 56962 macro avg 0.51 0.92 0.52 56962 weighted avg 1.00 0.95 0.97 56962 The metrics showed the following:\nThe model had a high recall, meaning it was good at detecting fraudulent transactions, but it also had a low precision, meaning there were many false positives. High False Positive Rate: Many normal transactions were incorrectly flagged as fraud, which indicates that the threshold may need further fine-tuning to reduce these false positives. Improving the Model # To improve the model’s performance, we could try:\nAdjusting the Threshold: Trying different percentile values or using a validation dataset to determine the best threshold value. Model Complexity: Adding more neurons or layers, or even using a Variational Autoencoder (VAE) for better results. Resampling the Data: To handle the imbalance in the dataset, we could use oversampling for the fraud class to help the model learn these patterns better. Video # ","date":"21 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_21/","section":"Challenges","summary":"Today, we continued with our autoencoder-based anomaly detection by fine-tuning the model and evaluating its performance on the Credit Card Fraud Detection Dataset.","title":"Day 21: Fine-Tune and Evaluate Autoencoder Model for Anomaly Detection","type":"challenge"},{"content":"Today, we began building an autoencoder-based anomaly detection system using the Credit Card Fraud Detection Dataset from Kaggle. Our main goal for today was to set up the data and build an initial version of the autoencoder model to detect anomalies. Fine-tuning and evaluation will be covered tomorrow.\nStep 1: Data Preparation # The first step was to prepare the data. Since the dataset deals with credit card fraud, it contains mostly normal transactions, with only a small percentage of fraudulent transactions, making it a highly imbalanced dataset.\nImport Necessary Libraries # We started by importing all the necessary libraries for data processing, model building, and visualization.\nimport numpy as np import pandas as pd import tensorflow as tf from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense import matplotlib.pyplot as plt Pandas and Numpy: Used for handling and manipulating data. TensorFlow/Keras: Used for building the autoencoder model. Matplotlib: Used for visualizing the training progress. StandardScaler: Used to normalize the data for better model performance. Load and Preprocess the Data # We loaded the credit card fraud dataset and normalized it for training the autoencoder.\n# Load the dataset data = pd.read_csv(\u0026#39;creditcard.csv\u0026#39;) # Display basic information print(data.head()) print(data.info()) Dataset Overview: The dataset includes features V1, V2, \u0026hellip;, V28, which are PCA-transformed features, as well as Time, Amount, and the target label Class (0 for normal transactions, 1 for fraud). Data Processing Steps # The data needs to be properly processed before feeding it into the autoencoder.\n# Extract features and labels X = data.drop(columns=[\u0026#39;Class\u0026#39;, \u0026#39;Time\u0026#39;]) y = data[\u0026#39;Class\u0026#39;] # Standardize the \u0026#39;Amount\u0026#39; column and other features scaler = StandardScaler() X_scaled = scaler.fit_transform(X) # Split the data: 80% for training, 20% for testing X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42) # Use only non-fraudulent data to train the autoencoder X_train_normal = X_train[y_train == 0] Feature Extraction: We used all features except Class and Time. Scaling: We used StandardScaler to scale features to have a mean of 0 and a standard deviation of 1. This is important to help the model converge during training. Data Splitting: We split the data into training (80%) and test (20%) sets. Train on Normal Data Only: We filtered only the normal transactions (y_train == 0) for training the autoencoder. The autoencoder needs to learn what normal data looks like, which is essential for detecting anomalies. Step 2: Set Up the Autoencoder Model # An autoencoder is made up of two main components:\nEncoder: Compresses the data into a smaller representation. Decoder: Attempts to reconstruct the original data from the compressed representation. Build the Autoencoder Model # # Build the autoencoder model model = Sequential() # Encoder layers model.add(Dense(14, activation=\u0026#39;relu\u0026#39;, input_shape=(X_train.shape[1],))) # First layer with 14 neurons model.add(Dense(7, activation=\u0026#39;relu\u0026#39;)) # Reduced to 7 neurons # Decoder layers model.add(Dense(14, activation=\u0026#39;relu\u0026#39;)) # Upsample back to 14 neurons model.add(Dense(X_train.shape[1], activation=\u0026#39;linear\u0026#39;)) # Final layer to reconstruct original input shape # Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;mse\u0026#39;) model.summary() Explanation # Encoder Part: The encoder has two layers that reduce the original feature space from 30 features to 7 features. This step captures the essential information while reducing noise. Decoder Part: The decoder tries to reconstruct the original features from the smaller representation. Linear Activation in the final layer is used because we want the model to output continuous real values to match the original input features. Loss Function: We used Mean Squared Error (MSE) as the loss function, which measures how well the model is reconstructing the input. Model Summary # The model summary gives an overview of the number of parameters and the layers used in the autoencoder.\nStep 3: Train the Autoencoder # The next step is to train the autoencoder on normal data only. This way, the model learns to reconstruct typical transaction patterns.\n# Train the autoencoder history = model.fit(X_train_normal, X_train_normal, epochs=50, batch_size=256, validation_split=0.2, verbose=1) Training on Normal Data: We train the autoencoder on only normal transactions to learn normal behavior patterns. Epochs and Batch Size: We used 50 epochs and a batch size of 256. The number of epochs is the number of complete passes through the training data, and batch size determines how many samples are processed before the model is updated. Validation Split: We used 20% of the training data for validation to monitor the model\u0026rsquo;s performance during training. Plot Training Loss # We used a loss plot to monitor how well the model is learning to reconstruct the normal transactions.\n# Plot the training and validation loss plt.plot(history.history[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss\u0026#39;) plt.plot(history.history[\u0026#39;val_loss\u0026#39;], label=\u0026#39;Validation Loss\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.legend() plt.title(\u0026#39;Training and Validation Loss for Autoencoder\u0026#39;) plt.show() Loss Plot: The training loss should ideally decrease over time, indicating that the model is improving. The validation loss helps to check if the model is overfitting (performing well on training data but poorly on validation data). Summary of Part 1: Data and Model Setup # We loaded and processed the credit card dataset, separating it into features and labels. The features were scaled for optimal performance. We built an autoencoder model that consists of an encoder and decoder. The encoder reduces the input to a latent space, and the decoder reconstructs the original features. The model was trained on only normal transactions to learn what normal patterns look like, which is key for detecting anomalies based on reconstruction errors. Next Steps for Part 2 (Tomorrow) # Fine-Tune the Model: Adjust hyperparameters and make improvements to the training process. Detect Anomalies: Use reconstruction error to classify transactions as either normal or fraudulent. Evaluate the Model: Assess model performance using metrics like Precision, Recall, and AUC to understand how well the model detects anomalies. With today’s progress, we are ready to take on anomaly detection and fine-tuning tomorrow.\nVideo # ","date":"20 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_20/","section":"Challenges","summary":"Today, we began building an autoencoder-based anomaly detection system using the Credit Card Fraud Detection Dataset from Kaggle. Our main goal for today was to set up the data and build an initial version of the autoencoder model to detect anomalies. Fine-tuning and evaluation will be covered tomorrow.","title":"Day 20: Building an Autoencoder-Based Anomaly Detection System (Part 1: Data and Model Setup)","type":"challenge"},{"content":"Welcome to Day 18 of our deep learning challenge! Today, we will explore the theory behind the attention mechanism and understand how it can be added to an LSTM model for machine translation. We’ll focus on building a foundation that will help us implement this mechanism in code tomorrow (Day 19).\nWhy Attention? # In machine translation, like translating text from English to French, an LSTM processes the entire input sentence and then tries to generate the translation. The problem is that, as the sequence gets longer, the LSTM starts to forget parts of the information it received earlier. This makes it difficult for the LSTM to properly handle long sentences because it loses important details.\nThe attention mechanism helps solve this problem by allowing the model to focus on relevant parts of the input sentence at every step of the output generation. Essentially, it’s like the model asking itself, “Which words in the input sentence are most important for generating the next word?” and then focusing on those specific parts.\nWhat is Attention Mechanism? # The attention mechanism is a method that allows the model to selectively focus on different parts of the input sequence. Instead of relying solely on the final output of an LSTM (which might be missing some important details), the attention mechanism assigns weights to each input word to decide how much attention should be paid to each word when producing the next word in the output.\nThink of it like reading a book and summarizing it: instead of trying to remember the entire book, you selectively look back at the important sections that are most relevant to the summary you are writing at that moment. This helps you make a better, context-aware summary.\nHow Does Attention Work in LSTM? # Attention works in three main steps:\nScoring: Calculate a score for each input word to determine how important it is for the current output word. Weighting: Use these scores to generate weights, which determine how much attention each word should receive. Context Vector: Use the weights to create a context vector, which is a weighted combination of all the input words. This context vector is then used along with the LSTM\u0026rsquo;s hidden state to predict the next word. Let’s break down these steps in a simpler way:\n1. Scoring the Input Words # The model computes a score for each word in the input sentence to determine its relevance for generating the current output word. This score is usually computed based on the current hidden state of the LSTM (i.e., what the LSTM already knows) and each of the input words.\nThink of this as the model deciding which parts of the input are important based on the current output that it’s generating. The score can be computed using a simple neural network layer that takes the LSTM\u0026rsquo;s hidden state and each input word as inputs and produces a score. Scoring Calculation\nTo understand how scores are calculated in an attention mechanism:\nThe score of each input word indicates how relevant that word is to the current output word that the model is generating. The score is often calculated by using a simple layer in the network, like taking the dot product between the current hidden state of the LSTM and the embedding of each word in the input. The dot product essentially measures similarity: the higher the dot product, the more similar the two vectors are. For example, imagine we have an input sentence with four words: [\u0026quot;I\u0026quot;, \u0026quot;am\u0026quot;, \u0026quot;learning\u0026quot;, \u0026quot;LSTMs\u0026quot;]. Let\u0026rsquo;s say our LSTM is trying to generate the next word in the translated output. The LSTM\u0026rsquo;s current hidden state is compared with the representation of each of the input words to get scores like [2.0, 0.5, 3.0, 1.5]. This means that the word “learning” has the highest score of 3.0, indicating it\u0026rsquo;s the most relevant at this point.\n2. Generating Weights # The scores are then normalized using a technique called softmax. Softmax converts these scores into values between 0 and 1, such that all the values add up to 1. These values are called attention weights.\nThe higher the weight, the more attention that word will receive. The softmax function makes sure that the weights are easy to interpret as probabilities, which helps the model decide how much each word contributes to generating the current output word. Generating Weights Calculation\nOnce we have the scores for each input word, we pass them through a softmax function to convert them into attention weights. These weights are values between 0 and 1 that add up to 1. For example, if we have the scores [2.0, 0.5, 3.0, 1.5], applying softmax might give us weights like [0.25, 0.10, 0.45, 0.20]. 3. Creating the Context Vector # The model then uses these attention weights to compute a weighted sum of all the input words. This weighted sum is called the context vector.\nThe context vector is essentially a summary of all the input words, but it focuses more on the words with higher attention weights. This context vector is then combined with the LSTM’s current hidden state to generate the final output for the current time step. Context Vector Example\nEach input word is represented by a vector of numbers called an embedding. Let\u0026rsquo;s say the embeddings are:\n\u0026ldquo;I\u0026rdquo;: [0.1, 0.2, 0.3, 0.4] \u0026ldquo;am\u0026rdquo;: [0.0, 0.1, 0.1, 0.1] \u0026ldquo;learning\u0026rdquo;: [0.4, 0.5, 0.5, 0.6] \u0026ldquo;LSTMs\u0026rdquo;: [0.3, 0.3, 0.2, 0.4] The context vector is calculated by taking a weighted sum of these embeddings based on their attention weights:\nFor \u0026ldquo;I\u0026rdquo;: 0.25 * [0.1, 0.2, 0.3, 0.4] = [0.025, 0.05, 0.075, 0.1] For \u0026ldquo;am\u0026rdquo;: 0.10 * [0.0, 0.1, 0.1, 0.1] = [0.0, 0.01, 0.01, 0.01] For \u0026ldquo;learning\u0026rdquo;: 0.45 * [0.4, 0.5, 0.5, 0.6] = [0.18, 0.225, 0.225, 0.27] For \u0026ldquo;LSTMs\u0026rdquo;: 0.20 * [0.3, 0.3, 0.2, 0.4] = [0.06, 0.06, 0.04, 0.08] Now, add all these weighted embeddings together:\n[0.025, 0.05, 0.075, 0.1] + [0.0, 0.01, 0.01, 0.01] + [0.18, 0.225, 0.225, 0.27] + [0.06, 0.06, 0.04, 0.08] = [0.265, 0.345, 0.35, 0.46] This final vector [0.265, 0.345, 0.35, 0.46] is called the context vector, and it serves as a summary of the input sentence with a focus on the important words. It will be used along with the LSTM’s current hidden state to generate the next word in the translated sentence.\nSimple Analogy: Attention in a Conversation # Imagine you are translating a long paragraph from English to French. Each time you translate a sentence, you might want to look back at specific words or phrases in the original paragraph. You don’t try to keep everything in your head at once—you selectively look back to find the parts that are most relevant to what you are currently translating.\nThe attention mechanism in an LSTM works similarly: it looks back at the input sequence and decides which parts are important for generating each word of the output. Attention in Machine Translation # In machine translation with LSTMs and attention:\nThe encoder reads the entire input sentence and produces a sequence of hidden states. At each step of the decoder (which generates the translated sentence), the attention mechanism helps decide which parts of the input sentence are most relevant to the current word being generated. This means the model doesn\u0026rsquo;t just rely on a single hidden state at the end of the input sentence—it uses information from all of the hidden states, focusing more on the most important ones. Types of Attention # There are a few common types of attention used in machine translation:\nGlobal Attention: The model looks at all input words when generating each output word. Local Attention: The model looks at a small subset of input words, which makes it more efficient and sometimes more accurate for longer sequences. Summary # Problem with LSTMs: LSTMs struggle with remembering long sequences because they have to store all the information in a single vector. Attention Mechanism: Allows the model to focus on relevant parts of the input sequence, dynamically deciding which parts to pay more attention to when generating each word. Key Steps: Calculate scores, generate attention weights, create a context vector. Usefulness: Attention makes LSTMs much better at handling long sentences and complex relationships in sequences by selectively remembering important information. Looking Forward to Day 19 # Tomorrow, we’ll implement this attention mechanism in code to build a complete machine translation model using LSTMs. We’ll see how we can integrate attention to make our LSTM more powerful and accurate for translating text.\nVideo # ","date":"18 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_18/","section":"Challenges","summary":"Welcome to Day 18 of our deep learning challenge! Today, we will explore the theory behind the attention mechanism and understand how it can be added to an LSTM model for machine translation. We’ll focus on building a foundation that will help us implement this mechanism in code tomorrow (Day 19).","title":"Day 18: Understanding Attention Mechanism for LSTM in Machine Translation","type":"challenge"},{"content":"On Day 19 of our deep learning journey, we tackled a complex but fascinating concept—adding an attention mechanism to an LSTM model for machine translation. Below, I\u0026rsquo;ll guide you step by step through the process of building this model and provide explanations for each part of the code to make everything clear and approachable.\nStep 1: Import Necessary Libraries # First, we import the essential libraries for data handling, model building, and training:\nimport numpy as np import tensorflow as tf from tensorflow.keras.layers import Input, Dot, LSTM, Dense, Embedding, Activation, Concatenate from tensorflow.keras.models import Model from tensorflow.keras.preprocessing.sequence import pad_sequences from tensorflow.keras.preprocessing.text import Tokenizer import tensorflow.keras.backend as K TensorFlow: Used to create and train the model. Dot, Concatenate: These layers help build the attention mechanism. Tokenizer and pad_sequences are used for text preprocessing. Step 2: Data Preprocessing # We define some parameters related to the sequences, such as maximum length and vocabulary size, and preprocess the data.\nmax_encoder_seq_length = 20 max_decoder_seq_length = 20 input_vocab_size = 10000 output_vocab_size = 10000 embedding_dim = 128 max_encoder_seq_length and max_decoder_seq_length: Define the maximum length of the input and output sequences. input_vocab_size and output_vocab_size: Vocabulary sizes for input (English) and output (French) sentences. embedding_dim: Embedding vector size for the input and output sequences. Tokenizer Setup # We initialize the tokenizers for both input and output sequences:\ninput_tokenizer = Tokenizer(num_words=input_vocab_size, filters=\u0026#39;\u0026#39;) output_tokenizer = Tokenizer(num_words=output_vocab_size, filters=\u0026#39;\u0026#39;) filters='' ensures that special tokens like \u0026lt;start\u0026gt; and \u0026lt;end\u0026gt; are not filtered out during tokenization. Texts and Tokenization # Next, we define our training sentences and tokenize them:\ninput_sequences = [\u0026#34;I am learning deep learning.\u0026#34;, \u0026#34;This is a test sentence.\u0026#34;] output_sequences = [\u0026#34;\u0026lt;start\u0026gt; Je suis en train d\u0026#39;apprendre l\u0026#39;apprentissage profond. \u0026lt;end\u0026gt;\u0026#34;, \u0026#34;\u0026lt;start\u0026gt; Ceci est une phrase de test. \u0026lt;end\u0026gt;\u0026#34;] input_tokenizer.fit_on_texts(input_sequences) output_tokenizer.fit_on_texts(output_sequences) input_sequences = input_tokenizer.texts_to_sequences(input_sequences) output_sequences = output_tokenizer.texts_to_sequences(output_sequences) input_sequences = pad_sequences(input_sequences, maxlen=max_encoder_seq_length, padding=\u0026#39;post\u0026#39;) output_sequences = pad_sequences(output_sequences, maxlen=max_decoder_seq_length, padding=\u0026#39;post\u0026#39;) \u0026lt;start\u0026gt; and \u0026lt;end\u0026gt; tokens are added to the output sequences to help the model know where the output begins and ends. Padding ensures that all sequences have the same length, which is necessary for batch processing. Step 3: Define the Model Components # Encoder # The encoder takes the input sequence and produces a series of hidden states and the final states:\nencoder_inputs = Input(shape=(max_encoder_seq_length,)) encoder_embedding = Embedding(input_dim=input_vocab_size, output_dim=embedding_dim)(encoder_inputs) encoder_lstm = LSTM(128, return_sequences=True, return_state=True) encoder_outputs, state_h, state_c = encoder_lstm(encoder_embedding) Input defines the placeholder for the input data. Embedding converts input words into dense vector representations. LSTM processes these embeddings, returning hidden states (encoder_outputs) for each time step and the final states (state_h, state_c). These final states are used to initialize the decoder. Decoder # The decoder generates the output sequence by taking the encoder\u0026rsquo;s hidden states and using them to predict each word in the target sequence.\ndecoder_inputs_layer = Input(shape=(max_decoder_seq_length,)) decoder_embedding = Embedding(input_dim=output_vocab_size, output_dim=embedding_dim)(decoder_inputs_layer) decoder_lstm = LSTM(128, return_sequences=True, return_state=True) decoder_outputs, _, _ = decoder_lstm(decoder_embedding, initial_state=[state_h, state_c]) decoder_inputs_layer represents the inputs to the decoder. Embedding layer converts each token into a dense vector. LSTM layer produces outputs for each time step, as well as the updated hidden states. Step 4: Add Attention Mechanism # The attention mechanism allows the decoder to focus on different parts of the encoder\u0026rsquo;s output while generating each word:\nattention = Dot(axes=[2, 2])([decoder_outputs, encoder_outputs]) attention = Activation(\u0026#39;softmax\u0026#39;)(attention) context = Dot(axes=[2, 1])([attention, encoder_outputs]) decoder_combined_context = Concatenate(axis=-1)([context, decoder_outputs]) Dot calculates the similarity between encoder outputs and decoder outputs, providing attention scores. Activation('softmax') normalizes these scores, turning them into probabilities. Dot again uses these scores to compute a context vector as a weighted sum of the encoder outputs. Concatenate combines the context vector with the current decoder output. Step 5: Output Layer # The concatenated output is passed through a dense layer to generate predictions for each word in the output vocabulary:\ndecoder_dense = Dense(output_vocab_size, activation=\u0026#39;softmax\u0026#39;) decoder_output_final = decoder_dense(decoder_combined_context) Dense(output_vocab_size) is used to predict the next word\u0026rsquo;s probability distribution over the output vocabulary. Step 6: Training Model Definition # We define the final model and compile it:\nmodel = Model([encoder_inputs, decoder_inputs_layer], decoder_output_final) model.compile( optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;] ) model.summary() Model is built with both encoder and decoder inputs. sparse_categorical_crossentropy is used as the loss function because our targets are not one-hot encoded. Training the Model # We train the model with the preprocessed data:\nmodel.fit( [input_sequences, decoder_inputs], output_sequences, batch_size=64, epochs=10, validation_split=0.2 ) batch_size is set to 64, and we train for 10 epochs. validation_split of 0.2 keeps a portion of the data for validation. Step 7: Inference Models for Translation # After training, we need separate encoder and decoder models for inference (translation).\nEncoder Model for Inference # encoder_model = Model(encoder_inputs, [encoder_outputs, state_h, state_c]) This model takes the encoder input and produces the encoder outputs and final states, which are used to initialize the decoder. Decoder Model for Inference # decoder_state_input_h = Input(shape=(128,)) decoder_state_input_c = Input(shape=(128,)) encoder_output_input = Input(shape=(max_encoder_seq_length, 128)) decoder_embedding2 = Embedding(input_dim=output_vocab_size, output_dim=embedding_dim)(decoder_inputs_layer) decoder_outputs2, state_h2, state_c2 = decoder_lstm(decoder_embedding2, initial_state=[decoder_state_input_h, decoder_state_input_c]) attention2 = Dot(axes=[2, 2])([decoder_outputs2, encoder_output_input]) attention2 = Activation(\u0026#39;softmax\u0026#39;)(attention2) context2 = Dot(axes=[2, 1])([attention2, encoder_output_input]) decoder_combined_context2 = Concatenate(axis=-1)([context2, decoder_outputs2]) decoder_output_final2 = decoder_dense(decoder_combined_context2) decoder_model = Model( [decoder_inputs_layer, encoder_output_input, decoder_state_input_h, decoder_state_input_c], [decoder_output_final2, state_h2, state_c2] ) decoder_state_input_h and decoder_state_input_c represent the hidden and cell states fed into the LSTM at each step. This model is used iteratively to generate the translated output word by word. Video # ","date":"18 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_19/","section":"Challenges","summary":"On Day 19 of our deep learning journey, we tackled a complex but fascinating concept—adding an attention mechanism to an LSTM model for machine translation. Below, I’ll guide you step by step through the process of building this model and provide explanations for each part of the code to make everything clear and approachable.","title":"Day 19: Attention Mechanism for LSTM in Machine Translation","type":"challenge"},{"content":"Welcome to Day 17 of our deep learning challenge! Today, we will build an LSTM (Long Short-Term Memory) model for sentiment analysis using the IMDb movie reviews dataset. Sentiment analysis aims to determine whether the sentiment of a given movie review is positive or negative.\nWhy LSTMs? # LSTMs are a special type of Recurrent Neural Network (RNN) that are especially good at learning from long-term dependencies. Unlike standard RNNs, LSTMs can remember information for longer periods, which makes them ideal for tasks involving text sequences, like sentiment analysis.\nStep-by-Step Solution # Step 1: Import Libraries # First, we import all the necessary libraries for working with data, building our model, and training it.\nimport numpy as np import tensorflow as tf from tensorflow.keras.datasets import imdb from tensorflow.keras.preprocessing.sequence import pad_sequences from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout import matplotlib.pyplot as plt imdb: Contains the IMDb dataset, a collection of 50,000 movie reviews labeled as positive or negative. pad_sequences: Used to make sure all input sequences are of the same length. Embedding, LSTM, Dense, Dropout: Components to build our LSTM model. Step 2: Load and Preprocess the IMDb Dataset # The IMDb dataset comes with pre-tokenized data. We need to load it and prepare it for training.\n# Load the IMDb dataset vocab_size = 10000 # Restricting the vocabulary to the 10,000 most common words (X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=vocab_size) # Padding sequences to ensure uniform length max_length = 200 # Set the maximum length for each review X_train = pad_sequences(X_train, maxlen=max_length, padding=\u0026#39;post\u0026#39;) X_test = pad_sequences(X_test, maxlen=max_length, padding=\u0026#39;post\u0026#39;) Explanation # vocab_size = 10000: We\u0026rsquo;re limiting the number of unique words to 10,000 for simplicity. imdb.load_data(num_words=vocab_size): Loads the dataset, including only the 10,000 most common words. pad_sequences: Since reviews have different lengths, we use pad_sequences to make them all 200 words long. Shorter reviews are padded with zeros, while longer reviews are truncated. Step 3: Define the LSTM Model # Now, we build the LSTM model.\n# Define the LSTM model model = Sequential() # Embedding layer to convert word indices into dense vectors embedding_dim = 128 model.add(Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=max_length)) # LSTM layer with 128 units model.add(LSTM(units=128, return_sequences=False)) # Adding a dropout layer to prevent overfitting model.add(Dropout(0.5)) # Output layer with a single neuron for binary classification model.add(Dense(1, activation=\u0026#39;sigmoid\u0026#39;)) # Summary of the model model.summary() Explanation # Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=max_length): Converts word indices to dense vectors of fixed length (embedding_dim=128). This layer learns an embedding representation for each word during training. LSTM(units=128): The LSTM layer has 128 units, which means it can learn complex relationships in the input sequences. Dropout(0.5): Adds dropout to prevent overfitting by randomly setting 50% of the units to zero during training. Dense(1, activation='sigmoid'): A single neuron with sigmoid activation for binary classification (positive or negative). Step 4: Compile the Model # We compile the model by specifying the optimizer, loss function, and metrics.\n# Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;binary_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) Explanation # optimizer='adam': We use the Adam optimizer, which adjusts the learning rate during training for faster convergence. loss='binary_crossentropy': Since we are dealing with a binary classification problem (positive vs. negative), we use binary cross-entropy as the loss function. metrics=['accuracy']: We evaluate the model using accuracy. Step 5: Train the Model # We train the model on the IMDb training dataset.\n# Train the model history = model.fit( X_train, y_train, epochs=5, batch_size=64, validation_split=0.2, verbose=1 ) Explanation # epochs=5: Training for 5 epochs is sufficient for this example to see the learning trends. batch_size=64: We process 64 samples per training step, which balances memory use and training speed. validation_split=0.2: Use 20% of the training data as a validation set to monitor the model\u0026rsquo;s performance. Step 6: Evaluate the Model # After training, we evaluate the model\u0026rsquo;s performance on the test dataset.\n# Evaluate the model on test data test_loss, test_accuracy = model.evaluate(X_test, y_test) print(f\u0026#34;Test Accuracy: {test_accuracy:.4f}\u0026#34;) Explanation # model.evaluate(X_test, y_test): Evaluates the model\u0026rsquo;s accuracy and loss on unseen test data to determine how well the model generalizes. Test Accuracy: Prints the accuracy on the test dataset. Step 7: Plot Training and Validation Loss # We plot the training and validation loss to understand how well the model learned.\n# Plotting training and validation loss plt.plot(history.history[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss\u0026#39;) plt.plot(history.history[\u0026#39;val_loss\u0026#39;], label=\u0026#39;Validation Loss\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss\u0026#39;) plt.legend() plt.show() Explanation # Plotting the Loss: This helps visualize whether the model is overfitting (when validation loss is much higher than training loss) or underfitting (both losses remain high). Summary of LSTM Sentiment Analysis Model # We built an LSTM model to classify movie reviews as positive or negative. The model contains an embedding layer to learn word representations, an LSTM layer to learn sequence patterns, and a dropout layer to prevent overfitting. The final output layer is a single neuron for binary classification. We used the Adam optimizer and trained the model for 5 epochs. The model was evaluated on a test dataset to determine its accuracy. Possible Improvements # Increase Vocabulary Size: Increasing vocab_size may lead to a richer representation of words, which could improve accuracy. Use Pre-trained Embeddings: Instead of learning embeddings from scratch, we could use pre-trained embeddings like GloVe to give the model a better starting point. Experiment with LSTM Layers: Adding more LSTM layers or units could help the model learn more complex relationships. Video # ","date":"17 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_17/","section":"Challenges","summary":"Welcome to Day 17 of our deep learning challenge! Today, we will build an LSTM (Long Short-Term Memory) model for sentiment analysis using the IMDb movie reviews dataset. Sentiment analysis aims to determine whether the sentiment of a given movie review is positive or negative.","title":"Day 17: Building an LSTM Model for Sentiment Analysis","type":"challenge"},{"content":"Welcome to Day 16 of our deep learning challenge! Today, we are building a basic Recurrent Neural Network (RNN) model for temperature forecasting using the data we prepared in Day 15. Below, I provide the complete code along with detailed explanations.\nIntroduction to RNNs # A Recurrent Neural Network (RNN) is a type of neural network specifically designed for working with sequential data. RNNs have the ability to remember previous information, which helps them make better predictions based on context. This characteristic is what makes RNNs ideal for time series forecasting, where past data points are used to predict future values. In our case, we\u0026rsquo;re using RNNs to predict future temperatures.\nHow Does an RNN Work? # An RNN processes each value in the sequence one step at a time, updating its memory (called the hidden state) at each step. Think of an RNN like reading a book: each sentence makes more sense if you remember what happened before. By looking at past temperatures, an RNN can learn trends and predict future temperatures. Full Code for Day 16: Building and Training the RNN Model # Below is the complete code for building, training, and evaluating a simple RNN model to predict future temperatures based on past observations.\nimport numpy as np import tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.layers import SimpleRNN, Dense import matplotlib.pyplot as plt # Step 1: Define the RNN Model model = Sequential() # Add a SimpleRNN layer model.add(SimpleRNN(units=50, activation=\u0026#39;tanh\u0026#39;, input_shape=(X_train.shape[1], 1))) # Add a Dense layer to output the prediction model.add(Dense(1)) # Summary of the model model.summary() Explanation # Importing Libraries: We import necessary libraries like NumPy, TensorFlow, and matplotlib to handle the data, build the model, and plot the training progress. Sequential(): This creates a simple sequential model, where we add layers one after another. SimpleRNN(units=50, activation='tanh', input_shape=(X_train.shape[1], 1)): This adds an RNN layer with: units=50: The number of neurons in the layer. More units mean more memory capacity. activation='tanh': The tanh activation function helps the RNN process sequences better by providing non-linearity. input_shape=(X_train.shape[1], 1): The shape of the input is defined as (sequence length, number of features). Here, we have one feature (temperature). Dense(1): The final layer is a fully connected layer that outputs 1 value (the predicted temperature). # Step 2: Compile the Model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;mse\u0026#39;, metrics=[\u0026#39;mae\u0026#39;]) Explanation # model.compile(): Compiles the RNN model to specify how it will be trained. optimizer='adam': The Adam optimizer is used to adjust the model\u0026rsquo;s parameters to minimize loss during training. loss='mse': The Mean Squared Error (MSE) loss function is used for regression tasks where we predict continuous values. metrics=['mae']: We use Mean Absolute Error (MAE) as an evaluation metric to measure the model’s prediction accuracy. # Step 3: Train the Model history = model.fit( train_dataset, validation_data=val_dataset, epochs=20, verbose=1 ) Explanation # model.fit(): Trains the RNN model using the training data. train_dataset: This contains the training sequences we prepared in the earlier step. validation_data=val_dataset: We use the validation dataset to evaluate the model\u0026rsquo;s performance after each epoch. epochs=20: We train the model for 20 epochs, meaning it will iterate over the entire dataset 20 times. verbose=1: This shows detailed progress during training, which helps monitor learning. # Step 4: Evaluate the Model loss, mae = model.evaluate(test_dataset) print(f\u0026#34;Test Loss (MSE): {loss:.4f}\u0026#34;) print(f\u0026#34;Test Mean Absolute Error (MAE): {mae:.4f}\u0026#34;) Explanation # model.evaluate(): This evaluates the trained model on the test dataset. loss and mae give us a quantitative measure of how well the model has learned to predict temperatures on unseen data. # Step 5: Plot the Training History plt.plot(history.history[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss (MSE)\u0026#39;) plt.plot(history.history[\u0026#39;val_loss\u0026#39;], label=\u0026#39;Validation Loss (MSE)\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Mean Squared Error\u0026#39;) plt.title(\u0026#39;Training and Validation Loss\u0026#39;) plt.legend() plt.show() Explanation # plt.plot(): Plots the training and validation loss over epochs. history.history['loss']: Tracks the training MSE loss. history.history['val_loss']: Tracks the validation MSE loss. Plotting the Loss: This helps visualize how the model is improving during training and can help detect issues like overfitting. Summary of the RNN Model # We built an RNN to predict future temperatures based on a sequence of past temperatures. The RNN uses 50 units to learn patterns in the time series data, and a Dense layer outputs the predicted temperature. The model is compiled with the Adam optimizer and trained for 20 epochs using Mean Squared Error loss. We used training, validation, and test datasets to ensure the model learns effectively and generalizes well. Possible Improvements # Experiment with LSTM or GRU Layers: Simple RNNs are limited in terms of memory. LSTMs (Long Short-Term Memory) or GRUs (Gated Recurrent Units) can learn longer-term dependencies more effectively. Hyperparameter Tuning: Adjust units, learning rates, batch size, etc., to improve model performance. Video # ","date":"16 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_16/","section":"Challenges","summary":"Welcome to Day 16 of our deep learning challenge! Today, we are building a basic Recurrent Neural Network (RNN) model for temperature forecasting using the data we prepared in Day 15. Below, I provide the complete code along with detailed explanations.","title":"Day 16: Building and Training the RNN Model for Temperature Forecasting","type":"challenge"},{"content":"Welcome to Day 15 of our deep learning challenge! Today, we are preparing a time series dataset for training an RNN model for temperature forecasting. Below, I provide the complete code along with detailed explanations.\nFull Code for Day 15: Preparing Time Series Data for Temperature Forecasting # Below is the code to prepare the time series dataset from Day 15, which will be used for training the RNN.\nimport numpy as np import pandas as pd import tensorflow as tf from sklearn.preprocessing import MinMaxScaler import matplotlib.pyplot as plt # Load Jena Climate Dataset (or similar dataset for temperature prediction) url = \u0026#39;https://raw.githubusercontent.com/jbrownlee/Datasets/master/daily-min-temperatures.csv\u0026#39; data = pd.read_csv(url) # Convert the Date column to datetime and set it as the index data[\u0026#39;Date\u0026#39;] = pd.to_datetime(data[\u0026#39;Date\u0026#39;]) data.set_index(\u0026#39;Date\u0026#39;, inplace=True) # Plot the temperature over time plt.figure(figsize=(10, 6)) plt.plot(data, label=\u0026#39;Daily Min Temperature\u0026#39;) plt.xlabel(\u0026#39;Date\u0026#39;) plt.ylabel(\u0026#39;Temperature (°C)\u0026#39;) plt.title(\u0026#39;Daily Minimum Temperature Over Time\u0026#39;) plt.legend() plt.show() # Create time series windows def create_time_series_data(data, window_size, target_size=1): X, y = [], [] for i in range(len(data) - window_size - target_size + 1): X.append(data[i: i + window_size]) y.append(data[i + window_size: i + window_size + target_size]) return np.array(X), np.array(y) # Set window size for the time series WINDOW_SIZE = 30 # The past 30 days as input # Convert the temperature column into a numpy array temp_data = data[\u0026#39;Temp\u0026#39;].values # Create time series windows X, y = create_time_series_data(temp_data, WINDOW_SIZE) # Split the data into training, validation, and test sets (70%, 20%, 10%) train_size = int(len(X) * 0.7) val_size = int(len(X) * 0.2) X_train, y_train = X[:train_size], y[:train_size] X_val, y_val = X[train_size:train_size + val_size], y[train_size:train_size + val_size] X_test, y_test = X[train_size + val_size:], y[train_size + val_size:] # Normalize the data scaler = MinMaxScaler(feature_range=(0, 1)) # Fit the scaler on the training data and transform the training, validation, and test data X_train = scaler.fit_transform(X_train.reshape(-1, 1)).reshape(X_train.shape) X_val = scaler.transform(X_val.reshape(-1, 1)).reshape(X_val.shape) X_test = scaler.transform(X_test.reshape(-1, 1)).reshape(X_test.shape) # Targets (y) are also scaled to (0, 1) range y_train = scaler.transform(y_train) y_val = scaler.transform(y_val) y_test = scaler.transform(y_test) # Create TensorFlow datasets BATCH_SIZE = 32 BUFFER_SIZE = 1000 # Create training dataset object train_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train)) train_dataset = train_dataset.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE) # Create validation dataset object val_dataset = tf.data.Dataset.from_tensor_slices((X_val, y_val)) val_dataset = val_dataset.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE) # Create test dataset object test_dataset = tf.data.Dataset.from_tensor_slices((X_test, y_test)) test_dataset = test_dataset.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE) Explanation of Dataset Preparation Steps # Loading the Data: We load the Jena Climate dataset and convert the Date column to a datetime format, making it easier to work with time series data. Create Sliding Windows: We create sequences of length WINDOW_SIZE (e.g., past 30 days) to be used as inputs to predict the target value (y), which is the next temperature value. Splitting the Data: The dataset is split into training (70%), validation (20%), and testing (10%) to evaluate the model’s performance. Normalization: We use the MinMaxScaler to normalize the data between 0 and 1 to help the RNN model converge faster. Batching and Prefetching: We create TensorFlow datasets for training, validation, and testing. We also use shuffling, batching, and prefetching for efficient data handling during training. Cache: The .cache() method is used to store data in memory after it\u0026rsquo;s loaded the first time, making training faster since it avoids reloading from disk. Shuffle: .shuffle(BUFFER_SIZE) randomly shuffles the data, helping to prevent the model from learning any order-based biases from the data. Batch: .batch(BATCH_SIZE) groups the data into batches, which helps to make training more efficient by updating the model parameters less frequently but with more data at each step. Prefetch: .prefetch(tf.data.AUTOTUNE) allows TensorFlow to load the next batch while the current batch is being processed, speeding up training. This completes our Day 15 project of preparing the time series dataset for temperature forecasting. The prepared data is now ready for training an RNN model, which we will work on in Day 16.\nVideo # ","date":"15 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day_15/","section":"Challenges","summary":"Welcome to Day 15 of our deep learning challenge! Today, we are preparing a time series dataset for training an RNN model for temperature forecasting. Below, I provide the complete code along with detailed explanations","title":"Day 15: Preparing Time Series Data for Temperature Forecasting","type":"challenge"},{"content":" Knowledge Distillation: Building a Custom CNN-based Student Model Using a Pre-Trained Teacher Model # In this mini-project, we use Knowledge Distillation to train a smaller student CNN by learning from a larger, pre-trained teacher model. The teacher model helps the student model generalize better, enabling it to achieve comparable performance while being more efficient. Below, you will find a detailed breakdown of the Python code used to implement this process.\nStep-by-Step Code Explanation # import tensorflow as tf from tensorflow.keras.applications import ResNet50 from tensorflow.keras.layers import GlobalAveragePooling2D, Dense from tensorflow.keras.models import Model from tensorflow.keras.datasets import cifar10 from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dropout import tensorflow.keras.backend as K from tensorflow.keras.losses import categorical_crossentropy Import Libraries: Here, we import the necessary libraries. We use TensorFlow and Keras to handle deep learning models. We also import ResNet50 as the pre-trained teacher model, along with several layers for building our custom models. We use cifar10 as our dataset, and backend (K) is used to implement the custom loss function. Step 1: Load and Prepare the Teacher Model # # Load the ResNet50 pre-trained larger model (Teacher) teacher_model = ResNet50(weights=\u0026#39;imagenet\u0026#39;, include_top=False, input_shape=(128, 128, 3)) # Add custom classification layers on top x = teacher_model.output x = GlobalAveragePooling2D()(x) x = Dense(256, activation=\u0026#39;relu\u0026#39;)(x) predictions = Dense(10, activation=\u0026#39;softmax\u0026#39;)(x) # Final Teacher Model teacher_model = Model(inputs=teacher_model.inputs, outputs=predictions) Teacher Model Setup: We use ResNet50 as the teacher model, which is pre-trained on ImageNet. include_top=False: This excludes the default fully connected layers, allowing us to add custom layers. Custom Layers: After obtaining the feature maps from ResNet50, we add: GlobalAveragePooling2D: Reduces the dimensionality of the feature maps. Dense Layer (256 units): A dense layer with ReLU activation. Dense Layer (10 units, softmax): Outputs probabilities for 10 classes. # Compile the teacher model teacher_model.compile( optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;] ) Compile Teacher Model: The teacher model is compiled using the Adam optimizer, sparse categorical cross-entropy as the loss, and accuracy as the metric. Step 2: Load and Preprocess the Dataset # # Load the dataset (X_train, y_train), (X_val, y_val) = cifar10.load_data() # Normalize the pixel values between 0-1 X_train = X_train / 255.0 X_val = X_val / 255.0 # Resize the images to 128x128 to match the teacher\u0026#39;s input shape X_train = tf.image.resize(X_train, (128, 128)) X_val = tf.image.resize(X_val, (128, 128)) Dataset Preparation: The CIFAR-10 dataset is loaded, which contains 32x32 pixel images. Normalization: The images are normalized to have values between 0-1. Resizing: Since ResNet50 requires input dimensions of 128x128, we resize the images using tf.image.resize(). Step 3: Train the Teacher Model # # Train the teacher model. teacher_model.fit( X_train, y_train, validation_data=(X_val, y_val), epochs=10, batch_size=32, verbose=1, ) teacher_output = teacher_model.predict(X_val) Train Teacher Model: The teacher model is trained for 10 epochs with a batch size of 32. After training, we compute the teacher output (predictions on validation data), which will be used to guide the student model. Step 4: Define the Student Model # # Define the simple smaller CNN as student model student_model = Sequential() student_model.add(Dense(16, (3, 3), activation=\u0026#39;relu\u0026#39;, input_shape=(128, 128, 3))) student_model.add(MaxPooling2D(3, 3)) student_model.add(Dense(32, (3, 3), activation=\u0026#39;relu\u0026#39;)) student_model.add(MaxPooling2D(3, 3)) student_model.add(Dense(64, (3, 3), activation=\u0026#39;relu\u0026#39;)) student_model.add(Flatten()) student_model.add(Dense(128, activation=\u0026#39;relu\u0026#39;)) student_model.add(Dropout(0.5)) student_model.add(Dense(10, activation=\u0026#39;softmax\u0026#39;)) student_model.summary() Student Model Definition: The student model is defined to be simpler and smaller than the teacher model. It consists of multiple convolutional layers with MaxPooling to reduce dimensionality and a Dropout layer to avoid overfitting. Dense(10, activation='softmax'): The output layer predicts probabilities for 10 classes. Step 5: Define Distillation Loss # def distillation_loss(org_prediction, student_prediction, teachers_output, temperature=0.3, alpha=0.5): # Calculate the student loss using standard categorical cross entropy student_loss = categorical_crossentropy(org_prediction, student_prediction) # Student, Teacher soft target distribution teachers_soft = K.softmax(teachers_output / temperature) student_soft = K.softmax(student_prediction / temperature) distillation_loss = K.sum(teachers_soft * K.log(teachers_soft / student_soft)) # Combined loss: Kullback-Leibler divergence return alpha * student_loss + (1 - alpha) * distillation_loss Distillation Loss Function: This function calculates the combined loss for training the student model. student_loss: The standard categorical cross-entropy between the true labels and the student’s predictions. Soft Targets: The teacher’s output is softened using a temperature parameter. This helps convey more nuanced information to the student. Distillation Loss: Calculated using the Kullback-Leibler divergence between the teacher’s softened output and the student’s output. Combined Loss: Combines the student loss with the distillation loss, weighted by alpha. Step 6: Compile and Train the Student Model # # Compile the student model student_model.compile( optimizer=\u0026#39;adam\u0026#39;, loss=lambda org_prediction, student_prediction: distillation_loss(org_prediction, student_prediction, teacher_output), metrics=[\u0026#39;accuracy\u0026#39;] ) # Train the model student_model.fit( X_train, y_train, validation_data=(X_val, y_val), epochs=10, batch_size=32, verbose=1 ) Compile Student Model: The student model is compiled using the custom distillation loss function defined earlier. The lambda function ensures that the distillation loss uses both the original predictions and the teacher\u0026rsquo;s output. Train Student Model: The student model is trained with 10 epochs and a batch size of 32. During training, it learns both from the true labels and from the teacher model\u0026rsquo;s predictions to improve generalization. Video # ","date":"14 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-14/","section":"Challenges","summary":"In this mini-project, we use Knowledge Distillation to train a smaller student CNN by learning from a larger, pre-trained teacher model. The teacher model helps the student model generalize better, enabling it to achieve comparable performance while being more efficient. Below, you will find a detailed breakdown of the Python code used to implement this process.","title":"Day 14: Building a Custom CNN-based Student Model Using a Pre-Trained Teacher Model","type":"challenge"},{"content":"Day 13 brings a deep dive into image segmentation with U-Net, a powerful neural network architecture for segmentation tasks. The Carvana dataset is perfect for this as it involves segmentation of car images, which makes the task visually interesting and a great learning experience.\nLet\u0026rsquo;s break down the task into easy-to-follow parts and help you implement it step by step!\nOverview of Image Segmentation and U-Net # Image Segmentation: Image segmentation is the process of labeling every pixel in an image such that different parts of the image are identified. For instance, you might want to segment cars from the background so that each pixel belongs to either the car or the background.\nU-Net: U-Net is a popular convolutional neural network (CNN) used for image segmentation. It is named \u0026ldquo;U-Net\u0026rdquo; due to its U-shaped architecture, consisting of:\nContracting Path: This is like an encoder, where features are extracted and the image resolution decreases. Expanding Path: This is like a decoder, where the spatial resolution is gradually restored to produce a segmentation map. The Carvana dataset is a dataset of car images that come with corresponding masks indicating which parts of the image contain the car, allowing you to train a model to segment cars from their backgrounds.\nSteps to Implement U-Net for Car Segmentation # Set Up Environment and Import Required Libraries Load and Preprocess the Dataset Build the U-Net Model Compile and Train the Model Evaluate the Model and Visualize Results Let’s implement these steps.\nStep 1: Set Up Environment and Import Libraries # First, make sure to install the required packages:\npip install tensorflow opencv-python matplotlib Pillow Then, let’s import the required libraries.\nimport os import cv2 import numpy as np import matplotlib.pyplot as plt from PIL import Image import tensorflow as tf from tensorflow.keras.layers import Conv2D, MaxPooling2D, UpSampling2D, concatenate, Input from tensorflow.keras.models import Model Explanation:\nWe use OpenCV for image processing, NumPy for numerical operations, Matplotlib for plotting, and TensorFlow/Keras for building the U-Net model. PIL (Pillow) is used to handle image formats, especially for masks in GIF format. Step 2: Load and Preprocess the Dataset # Here, we will load both the images and their corresponding masks from the dataset folders. We will resize them, normalize them, and make them ready for training.\n# Set paths to the dataset folders IMAGE_DIR = \u0026#34;dataset/carvana/train/\u0026#34; # Path to the train folder MASK_DIR = \u0026#34;dataset/carvana/train_masks/\u0026#34; # Path to the train_masks folder def load_data(image_dir, mask_dir, image_size=(128, 128)): images = [] masks = [] # Load images and masks for image_name in os.listdir(image_dir): # Skip hidden files or irrelevant files if any if image_name.startswith(\u0026#39;.\u0026#39;): continue # Construct paths to image and corresponding mask img_path = os.path.join(image_dir, image_name) # Mask file has \u0026#34;_mask\u0026#34; appended before the extension base_name = image_name.replace(\u0026#34;.jpg\u0026#34;, \u0026#34;\u0026#34;) mask_name = f\u0026#34;{base_name}_mask.gif\u0026#34; mask_path = os.path.join(mask_dir, mask_name) # Load the image using OpenCV img = cv2.imread(img_path) if img is None: print(f\u0026#34;Warning: Image {img_path} not found or couldn\u0026#39;t be loaded.\u0026#34;) continue img = cv2.resize(img, image_size) / 255.0 # Resize and normalize image to [0, 1] # Load the mask using PIL (Pillow) try: mask = Image.open(mask_path) mask = mask.convert(\u0026#39;L\u0026#39;) # Convert to grayscale mask = np.array(mask) # Convert to numpy array mask = cv2.resize(mask, image_size) # Resize the mask to the same size as the input image mask = mask / 255.0 # Normalize to range [0, 1] mask = np.expand_dims(mask, axis=-1) # Add channel dimension except Exception as e: print(f\u0026#34;Warning: Mask {mask_path} not found or couldn\u0026#39;t be loaded. Error: {e}\u0026#34;) continue images.append(img) masks.append(mask) return np.array(images), np.array(masks) # Load the data X, y = load_data(IMAGE_DIR, MASK_DIR) print(\u0026#34;Dataset loaded successfully.\u0026#34;) print(f\u0026#34;Images shape: {X.shape}, Masks shape: {y.shape}\u0026#34;) # Plot a few images and their corresponding masks plt.figure(figsize=(12, 6)) for i in range(3): plt.subplot(2, 3, i + 1) plt.imshow(X[i]) plt.title(\u0026#34;Car Image\u0026#34;) plt.axis(\u0026#39;off\u0026#39;) plt.subplot(2, 3, i + 4) plt.imshow(y[i].squeeze(), cmap=\u0026#39;gray\u0026#39;) plt.title(\u0026#34;Mask\u0026#34;) plt.axis(\u0026#39;off\u0026#39;) plt.tight_layout() plt.show() Training Images are in JPEG format, and masks are in GIF format. Masks have filenames that append \u0026ldquo;_mask\u0026rdquo; before the file extension. Explanation:\nLoading Images: Uses OpenCV to read images, resizes them to 128x128, and normalizes them to [0, 1]. Loading Masks: Uses Pillow to load GIF masks, convert them to grayscale, resize, normalize, and add a channel dimension to make them compatible for training. Step 3: Build the U-Net Model # The U-Net model has two main parts: a contracting path (encoder) and an expanding path (decoder).\ndef unet_model(input_size=(128, 128, 3)): inputs = Input(input_size) # Contracting Path (Encoder) c1 = Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(inputs) c1 = Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(c1) p1 = MaxPooling2D((2, 2))(c1) c2 = Conv2D(128, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(p1) c2 = Conv2D(128, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(c2) p2 = MaxPooling2D((2, 2))(c2) c3 = Conv2D(256, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(p2) c3 = Conv2D(256, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(c3) p3 = MaxPooling2D((2, 2))(c3) # Bottleneck c4 = Conv2D(512, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(p3) c4 = Conv2D(512, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(c4) # Expanding Path (Decoder) u5 = UpSampling2D((2, 2))(c4) u5 = concatenate([u5, c3]) c5 = Conv2D(256, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(u5) c5 = Conv2D(256, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(c5) u6 = UpSampling2D((2, 2))(c5) u6 = concatenate([u6, c2]) c6 = Conv2D(128, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(u6) c6 = Conv2D(128, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(c6) u7 = UpSampling2D((2, 2))(c6) u7 = concatenate([u7, c1]) c7 = Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(u7) c7 = Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;, padding=\u0026#39;same\u0026#39;)(c7) outputs = Conv2D(1, (1, 1), activation=\u0026#39;sigmoid\u0026#39;)(c7) model = Model(inputs, outputs) return model # Instantiate the model model = unet_model() model.summary() Explanation:\nContracting Path: Uses \u0026quot;Conv2D\u0026quot; and \u0026quot;MaxPooling2D\u0026quot; layers to extract features and downsample the image. Expanding Path: Uses \u0026quot;UpSampling2D\u0026quot; and \u0026quot;Concatenate\u0026quot; to reconstruct the segmentation map at higher resolution. outputs: Uses a \u0026quot;Conv2D\u0026quot; layer with a \u0026quot;1x1\u0026quot; kernel to produce the segmentation mask. Step 4: Compile and Train the Model # We will use the Adam optimizer and binary cross-entropy loss since this is a binary segmentation task (car vs. background).\n# Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;binary_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) # Train the model history = model.fit(X, y, validation_split=0.1, epochs=10, batch_size=8) Explanation:\n\u0026quot;optimizer='adam'\u0026quot;: Efficient optimization method for training. \u0026quot;loss='binary_crossentropy'\u0026quot;: Suitable for binary segmentation. \u0026quot;validation_split=0.1\u0026quot;: Uses 10% of the data for validation during training. Step 5: Evaluate the Model and Visualize Results # Finally, we’ll visualize how well the model has learned to segment the cars.\n# Select a sample from validation data to predict sample_image = X[0] sample_mask = y[0] # Expand dimensions to make it compatible with model input sample_image_expanded = np.expand_dims(sample_image, axis=0) # Predict mask predicted_mask = model.predict(sample_image_expanded)[0] # Plot original image, true mask, and predicted mask plt.figure(figsize=(15, 5)) plt.subplot(1, 3, 1) plt.title(\u0026#34;Original Image\u0026#34;) plt.imshow(sample_image) plt.subplot(1, 3, 2) plt.title(\u0026#34;True Mask\u0026#34;) plt.imshow(sample_mask.squeeze(), cmap=\u0026#39;gray\u0026#39;) plt.subplot(1, 3, 3) plt.title(\u0026#34;Predicted Mask\u0026#34;) plt.imshow(predicted_mask.squeeze(), cmap=\u0026#39;gray\u0026#39;) plt.show() Explanation:\nPrediction: Uses the trained model to predict the mask for a given image. Visualization: Compares the original image, true mask, and predicted mask side-by-side for easy evaluation. Video # ","date":"13 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-13/","section":"Challenges","summary":"Day 13 brings a deep dive into image segmentation with U-Net, a powerful neural network architecture for segmentation tasks. The Carvana dataset is perfect for this as it involves segmentation of car images, which makes the task visually interesting and a great learning experience.","title":"Day 13: Explore image segmentation with U-Net on Carvana dataset","type":"challenge"},{"content":"Day 12\u0026rsquo;s challenge is about implementing YOLO (You Only Look Once) for object detection. YOLO is one of the most well-known and widely used real-time object detection models due to its efficiency and speed.\nSince this is your first time diving into YOLO, we\u0026rsquo;ll take a tutorial-based approach and focus on understanding the core concepts and getting a simplified implementation up and running.\nOverview of YOLO (You Only Look Once) # YOLO is a popular object detection model that can identify and locate multiple objects in a single image with a single forward pass. It works by dividing an image into a grid and predicting bounding boxes and class probabilities for each section of the grid. YOLOv3 and YOLOv4 are widely used versions, but YOLOv5 is the most approachable for beginners due to its simplicity and open-source implementation. Objective # Use a pre-trained YOLO model to perform object detection. Instead of training YOLO from scratch (which requires substantial computational power), we’ll use a pre-trained model and run it on a sample image or video. Steps to Implement YOLO Using a Tutorial Approach # We’ll use YOLOv3 or YOLOv5, depending on simplicity and available resources. The easiest way is to use a pre-trained model and perform inference on sample images. We’ll use the OpenCV library with a pre-trained YOLOv3 model.\nStep 1: Set Up the Environment # First, let\u0026rsquo;s set up the environment. We will need:\nOpenCV to load and display images. YOLO weights and configuration files. Python packages like NumPy for general processing. You can install the required packages using:\npip install opencv-python-headless numpy Step 2: Download YOLO Weights and Configuration # YOLOv3 uses two main files:\nWeights file (\u0026ldquo;yolov3.weights\u0026rdquo;) contains the pre-trained parameters. Configuration file (\u0026ldquo;yolov3.cfg\u0026rdquo;) describes the architecture of the YOLO model. You can download them from the following sources:\nyolov3.weights: Download yolov3.cfg: Download Additionally, you’ll need the COCO dataset class names (\u0026ldquo;coco.names\u0026rdquo;) file, which contains the names of the 80 classes that YOLOv3 is trained to detect:\ncoco.names: Download Step 3: Implementing YOLO for Object Detection # Using OpenCV and a pre-trained YOLOv3 model, we’ll perform object detection on an image.\nFull Python Code\nimport cv2 import numpy as np # Load YOLOv3 weights, configuration, and COCO class names weights_path = \u0026#34;yolov3.weights\u0026#34; config_path = \u0026#34;yolov3.cfg\u0026#34; names_path = \u0026#34;coco.names\u0026#34; # Load the COCO class names with open(names_path, \u0026#34;r\u0026#34;) as f: class_names = f.read().strip().split(\u0026#34;\\n\u0026#34;) # Load the YOLOv3 model net = cv2.dnn.readNetFromDarknet(config_path, weights_path) # Set the model to use CPU or CUDA if available (comment if only CPU is used) net.setPreferableBackend(cv2.dnn.DNN_BACKEND_OPENCV) net.setPreferableTarget(cv2.dnn.DNN_TARGET_CPU) # Load the input image image = cv2.imread(\u0026#34;sample.jpg\u0026#34;) # Replace \u0026#39;sample.jpg\u0026#39; with your image file path (height, width) = image.shape[:2] # Create a blob from the input image blob = cv2.dnn.blobFromImage(image, scalefactor=1/255.0, size=(608, 608), swapRB=True, crop=False) net.setInput(blob) # Get the output layer names layer_names = net.getLayerNames() output_layer_names = [layer_names[i[0] - 1] for i in net.getUnconnectedOutLayers()] # Perform forward pass to get output from the output layers layer_outputs = net.forward(output_layer_names) # Initialize lists to hold the bounding boxes, confidences, and class IDs boxes = [] confidences = [] class_ids = [] # Loop over each output layer\u0026#39;s detections for output in layer_outputs: for detection in output: scores = detection[5:] # Get the scores for all classes class_id = np.argmax(scores) # Get the class ID with the highest score confidence = scores[class_id] # Get the highest score (confidence) if confidence \u0026gt; 0.5: # Filter out low confidence detections # Scale the bounding box back to the size of the image box = detection[0:4] * np.array([width, height, width, height]) (centerX, centerY, box_width, box_height) = box.astype(\u0026#34;int\u0026#34;) # Get the top-left corner coordinates x = int(centerX - (box_width / 2)) y = int(centerY - (box_height / 2)) # Save the box, confidence, and class ID boxes.append([x, y, int(box_width), int(box_height)]) confidences.append(float(confidence)) class_ids.append(class_id) # Apply Non-Maxima Suppression to suppress weak and overlapping bounding boxes indices = cv2.dnn.NMSBoxes(boxes, confidences, score_threshold=0.5, nms_threshold=0.4) # Draw the bounding boxes and class labels on the image for i in indices.flatten(): (x, y) = (boxes[i][0], boxes[i][1]) (w, h) = (boxes[i][2], boxes[i][3]) color = (0, 255, 0) # Green color for bounding box cv2.rectangle(image, (x, y), (x + w, y + h), color, 2) text = f\u0026#34;{class_names[class_ids[i]]}: {confidences[i]:.2f}\u0026#34; cv2.putText(image, text, (x, y - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2) # Show the output image cv2.imshow(\u0026#34;YOLO Object Detection\u0026#34;, image) cv2.waitKey(0) cv2.destroyAllWindows() Detailed Explanation of the Code # Load YOLO Configuration and Weights:\nnet = cv2.dnn.readNetFromDarknet(config_path, weights_path) This loads the YOLO configuration and weights files, initializing the network and preparing it for inference. Prepare the Input Image:\nblob = cv2.dnn.blobFromImage(image, scalefactor=1/255.0, size=(608, 608), swapRB=True, crop=False) net.setInput(blob) \u0026ldquo;blobFromImage()\u0026rdquo; converts the image into a blob, the format that YOLO expects. \u0026quot;scalefactor=1/255.0\u0026quot; normalizes pixel values to [0, 1]. The image is resized to (608, 608), which is the input size expected by YOLOv3. Perform Forward Pass:\nlayer_outputs = net.forward(output_layer_names) \u0026ldquo;forward()\u0026rdquo; performs a forward pass through the network, generating detections that include class scores and bounding boxes. Extract Information from the Detections:\nfor output in layer_outputs: for detection in output: scores = detection[5:] # Get the scores for all classes class_id = np.argmax(scores) # Get the class ID with the highest score confidence = scores[class_id] # Get the highest score (confidence) This part loops over all detections to extract class IDs, confidences, and bounding boxes for each detected object. A confidence threshold is applied to filter out weak detections. Non-Maxima Suppression (NMS):\nindices = cv2.dnn.NMSBoxes(boxes, confidences, score_threshold=0.5, nms_threshold=0.4) NMS removes overlapping bounding boxes for the same object, keeping only the box with the highest confidence. Draw Bounding Boxes:\ncv2.rectangle(image, (x, y), (x + w, y + h), color, 2) cv2.putText(image, text, (x, y - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2) Bounding boxes and class labels are drawn on the image.\nRunning the Code # To run the code successfully:\nPlace the following files in the same directory as your script: \u0026ldquo;yolov3.weights\u0026rdquo; \u0026ldquo;yolov3.cfg\u0026rdquo; \u0026ldquo;coco.names\u0026rdquo; An image file (e.g., \u0026ldquo;sample.jpg\u0026rdquo;) for testing. After running the script, you should see a window displaying the image with detected objects and bounding boxes.\nVideo # ","date":"12 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-12/","section":"Challenges","summary":"Day 12’s challenge is about implementing YOLO (You Only Look Once) for object detection. YOLO is one of the most well-known and widely used real-time object detection models due to its efficiency and speed.","title":"Day 12: Implementing YOLO for Object Detection","type":"challenge"},{"content":"Transfer learning is one of the most powerful techniques in deep learning, especially for computer vision tasks. With transfer learning, you leverage a pre-trained model (like VGG16) that has already learned features on a large dataset (such as ImageNet), and apply it to your own problem.\nWhat is Transfer Learning? # Transfer Learning involves taking a pre-trained model, which has already been trained on a large dataset, and fine-tuning it to perform a new task. This is useful when you have limited data or want to leverage the knowledge captured by a powerful model.\nVGG16 is a well-known pre-trained model from ImageNet that consists of 16 layers and is very effective at extracting useful features from images.\nOverview of Today\u0026rsquo;s Task # We will use the VGG16 model (pre-trained on ImageNet) and apply it to a new classification problem. You can choose a simple dataset, like Cats vs Dogs, or a small subset of CIFAR-10. We will freeze the convolutional base of VGG16 to use it as a feature extractor, then add custom fully connected layers on top for our classification task. Dataset # Download the images from Google APIs Storage and save them to your desired location. We will refer to this location later in the code.\nStep-by-Step Implementation # Step 1: Import Libraries and Load VGG16 # First, we need to import the necessary libraries and load the pre-trained VGG16 model from Keras.\nimport tensorflow as tf from tensorflow.keras.applications import VGG16 from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Flatten, Dropout from tensorflow.keras.preprocessing.image import ImageDataGenerator import matplotlib.pyplot as plt Explanation:\nVGG16 is a popular deep learning model used for image classification. It’s pre-trained on ImageNet, containing millions of images across 1000 classes. We’ll use VGG16 as a feature extractor and add custom layers for our classification task. Step 2: Load VGG16 Without the Top Layer # We’ll load the VGG16 model but exclude the top (fully connected) layers because we’ll be adding our own custom classification head.\n# Load VGG16 without the top layer (fully connected layers) vgg_base = VGG16(weights=\u0026#39;imagenet\u0026#39;, include_top=False, input_shape=(150, 150, 3)) # Freeze the convolutional base so that it\u0026#39;s not trained again vgg_base.trainable = False # Summary of the VGG16 base model vgg_base.summary() Explanation:\ninclude_top=False: This excludes the fully connected layers at the top of the network; we only want the convolutional base. input_shape=(150, 150, 3): This is the shape of the input images. We resize the input to 150x150 pixels with 3 color channels (RGB). Freezing the base: \u0026quot;vgg_base.trainable = False\u0026quot; freezes the convolutional base, so its weights are not updated during training. This way, we use the features VGG16 has already learned without retraining it from scratch. Step 3: Add Custom Layers to the VGG16 Base # We will add some custom layers to the convolutional base of VGG16 for our specific task.\n# Create a new model by adding custom layers to the VGG16 base model = Sequential() # Add the VGG16 base model.add(vgg_base) # Add a flattening layer to convert 3D output to 1D model.add(Flatten()) # Add fully connected layer with dropout for regularization model.add(Dense(256, activation=\u0026#39;relu\u0026#39;)) model.add(Dropout(0.5)) # Add output layer with softmax for classification (2 classes in this case) model.add(Dense(2, activation=\u0026#39;softmax\u0026#39;)) # Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) # Summary of the entire model model.summary() Explanation:\nFlatten Layer: \u0026quot;Flatten()\u0026quot; converts the 3D output from the VGG16 base to 1D so that it can be used in fully connected layers. Fully Connected Layer: \u0026quot;Dense(256, activation='relu')\u0026quot; adds a fully connected layer with 256 neurons and ReLU activation. \u0026quot;Dropout(0.5)\u0026quot; is used to prevent overfitting. Output Layer: \u0026quot;Dense(2, activation='softmax')\u0026quot; represents the output layer with 2 classes (e.g., cat vs. dog). Softmax is used for multi-class classification. Compile the Model: \u0026quot;Adam\u0026quot; optimizer is used to update weights, and categorical crossentropy is the loss function for classification. Step 4: Set Up Data Augmentation Using ImageDataGenerator # Since we need to feed our model with images, we’ll use ImageDataGenerator to preprocess and augment the data.\n# Set up data augmentation for training images train_datagen = ImageDataGenerator( rescale=1./255, # Normalize pixel values between 0 and 1 rotation_range=15, # Randomly rotate images by 15 degrees width_shift_range=0.1, # Randomly shift images horizontally height_shift_range=0.1,# Randomly shift images vertically zoom_range=0.1, # Randomly zoom in images horizontal_flip=True # Randomly flip images horizontally ) # Only rescale the validation images val_datagen = ImageDataGenerator(rescale=1./255) # Load data using flow_from_directory (you need a directory of images) train_generator = train_datagen.flow_from_directory( \u0026#39;path/to/train_data\u0026#39;, # Directory with training images target_size=(150, 150), # Resize all images to 150x150 batch_size=32, class_mode=\u0026#39;categorical\u0026#39; # Class mode for categorical labels ) validation_generator = val_datagen.flow_from_directory( \u0026#39;path/to/validation_data\u0026#39;, # Directory with validation images target_size=(150, 150), batch_size=32, class_mode=\u0026#39;categorical\u0026#39; ) Explanation:\nImageDataGenerator: Data Augmentation is applied to training images to increase variety. Validation images are only rescaled without augmentation for evaluation. flow_from_directory(): This function loads images from directories and applies transformations. \u0026quot;target_size=(150, 150)\u0026quot; resizes images to 150x150, with batch size set to 32. Step 5: Train the Model Using the Data Generators # Now that we have our model and data generators, we can train the model.\n# Train the model history = model.fit( train_generator, steps_per_epoch=train_generator.samples // train_generator.batch_size, validation_data=validation_generator, validation_steps=validation_generator.samples // validation_generator.batch_size, epochs=10, verbose=1 ) Explanation:\nTraining the Model: steps_per_epoch: This is the number of batches to run per epoch, calculated by dividing the total number of samples by batch size. validation_steps: The number of validation batches to run. epochs=10: Train the model for 10 epochs. Step 6: Evaluate and Visualize Training Performance # We can visualize the training history to understand the performance of our transfer learning model.\nimport pandas as pd # Convert the history to a DataFrame for easy visualization history_df = pd.DataFrame(history.history) # Plot training and validation accuracy plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;accuracy\u0026#39;], label=\u0026#39;Training Accuracy\u0026#39;) plt.plot(history_df[\u0026#39;val_accuracy\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Accuracy\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy\u0026#39;) plt.legend() plt.show() # Plot training and validation loss plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss\u0026#39;) plt.plot(history_df[\u0026#39;val_loss\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Loss\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss\u0026#39;) plt.legend() plt.show() Explanation:\nTraining vs Validation Accuracy/Loss: This helps visualize how well the model is performing during training. Ideally, training and validation accuracy should be close, indicating a well-generalized model. Key Concepts About Transfer Learning with VGG16 # Using Pre-trained Weights # The VGG16 model is trained on ImageNet, a large dataset with millions of images. It has learned to detect useful features such as edges, textures, and object parts. Instead of starting from scratch, you leverage this knowledge and apply it to your dataset. Feature Extraction # By freezing the convolutional base, you use the pre-learned features for your new classification task. Adding new fully connected layers allows you to classify new categories based on these features. Why Freeze the Layers? # If your dataset is small, you may not have enough data to train the deep layers of the VGG16 model effectively. Freezing the layers prevents them from being updated, so they act as a feature extractor. Expected Outcome # You should see that the model quickly learns and achieves relatively high accuracy, even with a small number of epochs. This is because VGG16 already has learned useful features, and you’re simply fine-tuning it for your specific task. Video # ","date":"11 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-11/","section":"Challenges","summary":"Day 11’s Transfer learning is one of the most powerful techniques in deep learning, especially for computer vision tasks. With transfer learning, you leverage a pre-trained model (like VGG16) that has already learned features on a large dataset (such as ImageNet), and apply it to your own problem.","title":"Day 11: Apply Transfer Learning with VGG16 for a simple classification task","type":"challenge"},{"content":"Day 10 is all about data augmentation, a great way to improve the performance and generalizability of your model. Let’s dive into the Fashion MNIST dataset and apply data augmentation using Keras.\nOverview of Data Augmentation # Data Augmentation is a technique used to artificially expand the size of a training dataset by applying random transformations like rotation, shifting, flipping, and zooming to the existing images. This helps improve the robustness of your model by exposing it to more variations, thereby reducing overfitting. Fashion MNIST: This dataset contains grayscale images of clothing items, with 10 classes like shirts, trousers, bags, etc. Goal of Today\u0026rsquo;s Task # Load the Fashion MNIST dataset. Build a CNN model. Use Keras’s pre-built data augmentation methods to generate more diverse training images. Train the model with augmented data to see the difference in performance. Step-by-Step Implementation # Step 1: Import Libraries and Load Dataset # We start by importing the necessary libraries and loading the Fashion MNIST dataset.\nimport tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense from tensorflow.keras.layers import Dropout from tensorflow.keras.datasets import fashion_mnist from tensorflow.keras.preprocessing.image import ImageDataGenerator import matplotlib.pyplot as plt # Load the Fashion MNIST dataset (X_train, y_train), (X_test, y_test) = fashion_mnist.load_data() # Display the shape of the data print(\u0026#39;Training data shape:\u0026#39;, X_train.shape) print(\u0026#39;Test data shape:\u0026#39;, X_test.shape) Explanation:\nFashion MNIST Dataset: Contains grayscale images of various clothing items, each with a shape of 28x28 pixels. Train and Test Split: The dataset is pre-split into training and test sets, making it easy to use. Step 2: Preprocess the Data # We need to normalize the images and reshape them for the model.\nExplanation:\nReshape: Adds an extra dimension for channels. Since the images are grayscale, there’s only 1 channel. Normalization: Scales pixel values between 0 and 1, which speeds up training and enhances model effectiveness. # Reshape the data to add the channel dimension (grayscale has 1 channel) X_train = X_train.reshape(X_train.shape[0], 28, 28, 1) X_test = X_test.reshape(X_test.shape[0], 28, 28, 1) # Normalize pixel values to be between 0 and 1 X_train = X_train.astype(\u0026#39;float32\u0026#39;) / 255.0 X_test = X_test.astype(\u0026#39;float32\u0026#39;) / 255.0 Step 3: Apply Data Augmentation Using ImageDataGenerator # We will use Keras’s ImageDataGenerator class to apply data augmentation.\n# Create an ImageDataGenerator with data augmentation settings datagen = ImageDataGenerator( rotation_range=15, # Rotate the image by up to 15 degrees width_shift_range=0.1, # Shift the width by up to 10% of the image width height_shift_range=0.1, # Shift the height by up to 10% of the image height zoom_range=0.1, # Zoom in by up to 10% horizontal_flip=True # Randomly flip images horizontally ) # Fit the ImageDataGenerator to the training data datagen.fit(X_train) # Let\u0026#39;s visualize some augmented images for X_batch, y_batch in datagen.flow(X_train, y_train, batch_size=9): # Create a grid of 3x3 images fig, ax = plt.subplots(3, 3, figsize=(8, 8)) for i in range(9): ax[i//3, i%3].imshow(X_batch[i].reshape(28, 28), cmap=\u0026#39;gray\u0026#39;) ax[i//3, i%3].axis(\u0026#39;off\u0026#39;) plt.suptitle(\u0026#39;Augmented Images\u0026#39;) plt.show() break # Only show one batch Explanation:\nImageDataGenerator: rotation_range=15: Rotates images randomly by up to 15 degrees. width_shift_range=0.1 and height_shift_range=0.1: Shifts images horizontally or vertically by 10% of the image dimensions. zoom_range=0.1: Randomly zooms in by up to 10%. horizontal_flip=True: Randomly flips images horizontally. datagen.flow(): Generates batches of augmented images. We visualize a 3x3 grid of augmented images for a better understanding of how augmentation works. Step 4: Define the CNN Model # We’ll create a CNN model to classify images in the Fashion MNIST dataset.\n# Define the CNN model model = Sequential() # Convolutional Layer 1 model.add(Conv2D(32, (3, 3), activation=\u0026#39;relu\u0026#39;, input_shape=(28, 28, 1))) model.add(MaxPooling2D((2, 2))) # Convolutional Layer 2 model.add(Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;)) model.add(MaxPooling2D((2, 2))) # Flatten the output to feed into fully connected layers model.add(Flatten()) # Fully Connected Layer model.add(Dense(128, activation=\u0026#39;relu\u0026#39;)) model.add(Dropout(0.5)) # Output Layer model.add(Dense(10, activation=\u0026#39;softmax\u0026#39;)) # Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) # Summary of the model model.summary() Explanation:\nConvolutional Layers: The first layer has 32 filters, and the second layer has 64 filters, enabling the model to learn different features. MaxPooling2D: Reduces the size of feature maps, allowing the model to focus on prominent features. Fully Connected Layer: A layer with 128 neurons, followed by a Dropout layer with a 50% dropout rate to prevent overfitting. Output Layer: Has 10 neurons (for the 10 classes in Fashion MNIST) with softmax activation to produce class probabilities. Step 5: Train the Model with Augmented Data # Now, we’ll train the model using the augmented images generated by ImageDataGenerator.\n# Train the model using the augmented data history = model.fit(datagen.flow(X_train, y_train, batch_size=64), validation_data=(X_test, y_test), epochs=15, verbose=1) Explanation:\ndatagen.flow(X_train, y_train, batch_size=64): Generates batches of augmented data on-the-fly during training. Validation Data: The test set is used for evaluation after each epoch. Epochs: The model is trained for 15 epochs, giving it enough time to learn from the augmented data. Step 6: Visualize Training Performance # We’ll plot the training and validation accuracy and loss to monitor the model’s performance over time.\nimport pandas as pd # Convert the history to a DataFrame for easy visualization history_df = pd.DataFrame(history.history) # Plot training and validation accuracy plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;accuracy\u0026#39;], label=\u0026#39;Training Accuracy\u0026#39;) plt.plot(history_df[\u0026#39;val_accuracy\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Accuracy\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy\u0026#39;) plt.legend() plt.show() # Plot training and validation loss plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss\u0026#39;) plt.plot(history_df[\u0026#39;val_loss\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Loss\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss\u0026#39;) plt.legend() plt.show() Explanation:\nTraining vs. Validation Accuracy/Loss: This helps identify if the model is overfitting or generalizing well to unseen data. Ideally, both accuracy metrics should increase, and both loss metrics should decrease. Key Points About Data Augmentation # More Data Without Collecting New Images # Data augmentation helps generate more diverse data from the existing dataset. You don’t need to collect additional images; you simply create variations from the ones you have. Improves Generalization # The model learns to recognize features even when they are rotated, shifted, flipped, or zoomed. This enhances the model’s ability to generalize and perform better on unseen data. Reduces Overfitting # Augmentation prevents the model from simply memorizing the training data by introducing more variety. As a result, the model learns robust patterns rather than memorizing specific examples. Video # ","date":"10 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-10/","section":"Challenges","summary":"Day 10 is all about data augmentation, a great way to improve the performance and generalizability of your model. Let’s dive into the Fashion MNIST dataset and apply data augmentation using Keras.","title":"Day 10: Data Augmentation with Fashion MNIST","type":"challenge"},{"content":" Overview of Today’s Task # Using the CIFAR-100 Dataset: This dataset is similar to CIFAR-10 but with 100 classes instead of 10. It’s more challenging due to the larger number of categories. Modifying the CNN: We’ll add more pooling layers to examine their impact on model performance. Visualizing Convolutional Filters: Understanding what features the CNN is learning helps us interpret its behavior, such as recognizing edges, textures, or abstract shapes in deeper layers. Step-by-Step Implementation # Step 1: Import Libraries and Load CIFAR-100 Dataset # import tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout from tensorflow.keras.datasets import cifar100 from tensorflow.keras.utils import to_categorical import matplotlib.pyplot as plt import numpy as np Explanation:\nCIFAR-100 Dataset: This dataset has 100 classes (e.g., flowers, insects, people) with 600 images per class. Step 2: Load and Preprocess the CIFAR-100 Dataset # # Load CIFAR-100 dataset (X_train, y_train), (X_test, y_test) = cifar100.load_data() # Normalize pixel values to be between 0 and 1 X_train = X_train.astype(\u0026#39;float32\u0026#39;) / 255.0 X_test = X_test.astype(\u0026#39;float32\u0026#39;) / 255.0 # Convert labels to categorical format (One-hot encoding) y_train = to_categorical(y_train, 100) y_test = to_categorical(y_test, 100) Explanation:\nNormalization: Adjusting pixel values between 0 and 1 enhances model performance. One-hot Encoding: Converts labels into a format suitable for multi-class classification, with each class represented by a binary vector. Step 3: Modify the CNN Model with Additional Pooling Layers # We’ll build a more complex CNN by adding extra pooling layers to explore their effect.\n# Define the modified CNN model model = Sequential() # First Convolutional Layer model.add(Conv2D(32, (3, 3), activation=\u0026#39;relu\u0026#39;, input_shape=(32, 32, 3))) model.add(MaxPooling2D((2, 2))) # Second Convolutional Layer model.add(Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;)) model.add(MaxPooling2D((2, 2))) # Third Convolutional Layer model.add(Conv2D(128, (3, 3), activation=\u0026#39;relu\u0026#39;)) model.add(MaxPooling2D((2, 2))) # Flatten the output to feed into fully connected layers model.add(Flatten()) # Fully Connected Layer model.add(Dense(128, activation=\u0026#39;relu\u0026#39;)) # Dropout Layer to avoid overfitting model.add(Dropout(0.5)) # Output Layer model.add(Dense(100, activation=\u0026#39;softmax\u0026#39;)) # Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) # Summary of the model model.summary() Explanation:\nThree Convolutional Layers: Each layer has an increasing number of filters (32, 64, 128) to capture progressively complex features. MaxPooling Layers: Placed after each convolutional layer, they reduce the spatial dimensions of feature maps, helping the network focus on essential features while reducing computational load. Fully Connected Layer and Dropout: A fully connected layer with 128 neurons, followed by a Dropout layer with a 50% dropout rate to prevent overfitting. Output Layer: 100 neurons represent each CIFAR-100 class, with softmax activation to output a probability distribution across classes. Step 4: Train the Model # # Train the model history = model.fit(X_train, y_train, epochs=20, batch_size=32, validation_data=(X_test, y_test), verbose=1) Explanation:\nEpochs: Training for 20 epochs to observe the model’s performance. Validation Data: Using test data during training to evaluate the model’s performance on unseen data. Step 5: Visualize Training Performance # We’ll plot the training and validation accuracy and loss to monitor the model’s learning over time.\nimport pandas as pd # Convert the history to a DataFrame for easy visualization history_df = pd.DataFrame(history.history) # Plot training and validation accuracy plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;accuracy\u0026#39;], label=\u0026#39;Training Accuracy\u0026#39;) plt.plot(history_df[\u0026#39;val_accuracy\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Accuracy\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy\u0026#39;) plt.legend() plt.show() # Plot training and validation loss plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss\u0026#39;) plt.plot(history_df[\u0026#39;val_loss\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Loss\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss\u0026#39;) plt.legend() plt.show() Explanation:\nTraining and Validation Accuracy/Loss: Observing these curves helps detect underfitting or overfitting. If training accuracy improves while validation accuracy doesn’t, it’s a sign the model may be overfitting. Step 6: Visualize Filters of the First Convolutional Layer # Visualizing the filters gives insight into what kind of features the CNN is learning, such as edges, colors, and textures.\n# Get weights of the first convolutional layer first_layer = model.layers[0] filters, biases = first_layer.get_weights() # Normalize filter values to 0-1 for visualization filters_min, filters_max = filters.min(), filters.max() filters = (filters - filters_min) / (filters_max - filters_min) # Plotting the filters n_filters = 6 # Number of filters to visualize fig, axes = plt.subplots(1, n_filters, figsize=(20, 5)) for i in range(n_filters): # Get the first channel of the filter (e.g., R channel) f = filters[:, :, 0, i] # Only plot the first channel for simplicity ax = axes[i] ax.imshow(f, cmap=\u0026#39;viridis\u0026#39;) # Plot the filter ax.axis(\u0026#39;off\u0026#39;) plt.suptitle(\u0026#39;Filters of the First Convolutional Layer\u0026#39;) plt.show() Explanation:\nFilters of the First Convolutional Layer: By examining the first layer’s filters (or weights), we can understand what patterns the network detects early on, such as edges or specific color gradients. Normalization: Filter values are normalized to a range of 0-1 for clear visualization by applying Min-Max normalization. Visualization: Using matplotlib, we plot the first few filters, each showing a different pattern that the network learns to identify, from basic shapes to textures. What Should You Expect to See?\nYou should see 6 subplots, each representing a filter. The patterns in the filters will vary, with different weights emphasizing different areas. For example, some filters might appear as dark on one side and light on the other — this means they are trying to detect edges in the images. Others might look like blotches of color, indicating that they are focusing on specific color patterns. Video # ","date":"9 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-9/","section":"Challenges","summary":"Today marks Day 8 of my 30 Days, 30 Deep Learning Projects Challenge. The task for today is to Modify CNN with pooling layers and visualize filters. Curious about how it went? Read on to see the results!","title":"Day 9: Modify CNN with pooling layers and visualize filters","type":"challenge"},{"content":" Today, we will build a simple CNN for image classification on the CIFAR-10 dataset. The CIFAR-10 dataset contains 60,000 color images of size 32x32 pixels, with 10 categories like airplanes, birds, cars, etc. We\u0026rsquo;ll use TensorFlow and Keras to build and train a Convolutional Neural Network (CNN), which is well-suited for handling image data due to its unique ability to capture spatial relationships in the data. What is a Convolutional Neural Network (CNN)? # A Convolutional Neural Network (CNN) is a type of deep learning model specially designed to work with images. CNNs can recognize patterns in images, much like how we use our eyes and brain to recognize faces, objects, and everything around us.\nThink of a CNN as a series of layers that each work together to identify features in an image, much like how our brain processes visual information step by step.\nImagine a Simple Example: Recognizing a Cat Picture # Imagine you’re looking at a picture of a cat. How do you know it’s a cat? Well, your brain processes the picture in parts. You might notice the whiskers, the eyes, the ears, and the shape of the face. Similarly, a CNN looks at the picture and breaks it down into parts to decide if it’s a cat.\nHow CNNs Work: # A CNN is composed of a series of layers, each working as a specialist to examine different parts of the image, each time getting more detailed. Here’s how it works (with cat example):\nInput Image: A picture of a cat is given to the CNN. Convolution Layer: The CNN applies filters to detect basic features, like edges or colors. Pooling Layer: The network simplifies the image, keeping only the essential information. Flattening: It turns the pooled image into a list of key features. Fully Connected Layer: The CNN examines the list of features and decides if it’s a cat, dog, bird, or something else. Read more for in depth understanding, with easy examples.\nStep-by-Step Guide to Build a Simple CNN for CIFAR-10 # Step 1: Import Necessary Libraries # We start by importing TensorFlow, Keras, and other helpful libraries.\nimport tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense from tensorflow.keras.layers import Dropout from tensorflow.keras.datasets import cifar10 from tensorflow.keras.utils import to_categorical import matplotlib.pyplot as plt Explanation:\nTensorFlow and Keras help us easily build and train the CNN. Conv2D, MaxPooling2D, Dropout, Flatten, Dense are layers used to build our CNN. cifar10 is the dataset we will work with. Step 2: Load and Preprocess the CIFAR-10 Dataset # We’ll load the CIFAR-10 dataset and prepare it for training.\n# Load CIFAR-10 dataset (X_train, y_train), (X_test, y_test) = cifar10.load_data() # Normalize pixel values to be between 0 and 1 X_train = X_train.astype(\u0026#39;float32\u0026#39;) / 255.0 X_test = X_test.astype(\u0026#39;float32\u0026#39;) / 255.0 # Convert labels to categorical format (One-hot encoding) y_train = to_categorical(y_train, 10) y_test = to_categorical(y_test, 10) Explanation:\nLoad Data: CIFAR-10 is already split into training and test sets. X_train contains the image data, and y_train contains the labels. Normalization: Pixel values are scaled between 0 and 1 to improve training. One-hot Encoding: Converts labels to one-hot format, needed for multi-class classification. Step 3: Define the CNN Model # We’ll build a simple CNN with two convolutional layers followed by fully connected layers for classification.\n# Define the CNN model model = Sequential() # First Convolutional Layer model.add(Conv2D(32, (3, 3), activation=\u0026#39;relu\u0026#39;, input_shape=(32, 32, 3))) model.add(MaxPooling2D((2, 2))) # Second Convolutional Layer model.add(Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;)) model.add(MaxPooling2D((2, 2))) # Flatten the output to feed into fully connected layers model.add(Flatten()) # Fully Connected Layer model.add(Dense(64, activation=\u0026#39;relu\u0026#39;)) # Dropout Layer to avoid overfitting model.add(Dropout(0.5)) # Output Layer model.add(Dense(10, activation=\u0026#39;softmax\u0026#39;)) # Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) # Summary of the model model.summary() Explanation:\nConv2D(32, (3, 3), activation=\u0026lsquo;relu\u0026rsquo;, input_shape=(32, 32, 3)): Applies 32 filters of size 3x3 with ReLU activation. The input shape is set to (32, 32, 3) for CIFAR-10 images. MaxPooling2D((2, 2)): Reduces spatial dimensions by taking the maximum value from each 2x2 region. Flatten(): Flattens 2D feature maps into a 1D vector for the fully connected layer. Dense(64, activation=\u0026lsquo;relu\u0026rsquo;): Adds a fully connected layer with 64 neurons. Dropout(0.5): Randomly drops 50% of neurons during training to prevent overfitting. Dense(10, activation=\u0026lsquo;softmax\u0026rsquo;): Output layer with 10 neurons for CIFAR-10 classes, using softmax for probability distribution. model.compile(): The model is compiled with the Adam optimizer, categorical cross-entropy loss, and accuracy as the metric. Step 4: Train the Model # Now, we’ll train the model on the CIFAR-10 dataset.\n# Train the model history = model.fit(X_train, y_train, epochs=20, batch_size=32, validation_data=(X_test, y_test), verbose=1) Explanation:\nepochs=20: The model trains for 20 epochs over the dataset. batch_size=32: 32 samples per gradient update. validation_data=(X_test, y_test): Enables evaluation on unseen data during training. Step 5: Evaluate the Model # We’ll evaluate the model on the test data to assess its performance.\n# Evaluate the model on the test set test_loss, test_accuracy = model.evaluate(X_test, y_test, verbose=2) print(f\u0026#39;Test accuracy: {test_accuracy:.2f}\u0026#39;) Explanation:\nmodel.evaluate(): Calculates the loss and accuracy on the test set. test_accuracy: Provides an estimate of how well the model generalizes to unseen data. Step 6: Visualize Training Performance # We can plot the training history to observe the model\u0026rsquo;s learning process over time.\n# Convert the history to a DataFrame for easy visualization import pandas as pd history_df = pd.DataFrame(history.history) # Plot training and validation accuracy plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;accuracy\u0026#39;], label=\u0026#39;Training Accuracy\u0026#39;) plt.plot(history_df[\u0026#39;val_accuracy\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Accuracy\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy\u0026#39;) plt.legend() plt.show() # Plot training and validation loss plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss\u0026#39;) plt.plot(history_df[\u0026#39;val_loss\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Loss\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss\u0026#39;) plt.legend() plt.show() Explanation:\nhistory_df[\u0026lsquo;accuracy\u0026rsquo;] and history_df[\u0026lsquo;val_accuracy\u0026rsquo;]: Shows how the model performs on the training and validation set over time. Training and Validation Loss: Helps identify if the model is overfitting by comparing training and validation loss. Video # ","date":"8 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-8/","section":"Challenges","summary":"Today marks Day 8 of my 30 Days, 30 Deep Learning Projects Challenge. The task for today is to Build a simple CNN for CIFAR-10 image classification. Curious about how it went? Read on to see the results!","title":"Day 8: Build a simple CNN for CIFAR-10 image classification","type":"challenge"},{"content":"Today we will explore hyperparameter tuning using the Keras Tuner, which is a powerful tool for optimizing the hyperparameters of a model to achieve the best possible performance. This will involve tuning different aspects of a neural network, like the number of neurons, learning rate, batch size, etc., to see how they impact performance.\nWhat is Hyperparameter Tuning? # Hyperparameters are settings that you need to define before training your model, such as the number of neurons in a layer, the learning rate of the optimizer, the batch size, and the number of layers. Hyperparameter tuning is the process of finding the best combination of these settings to maximize the model\u0026rsquo;s performance. Keras Tuner is a library that helps automate the process of trying different combinations of hyperparameters. Plan: # Install and import Keras Tuner. Define a simple neural network model for tuning. Use Keras Tuner to find the optimal hyperparameters. Train and evaluate the best model. We\u0026rsquo;ll create a small neural network and use the Keras Tuner to fine-tune the hyperparameters.\nStep-by-Step Implementation # Step 1: Install and Import Keras Tuner # First, if you don\u0026rsquo;t have Keras Tuner installed, you\u0026rsquo;ll need to install it:\npip install keras-tuner Next, import the necessary libraries.\nimport tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense from sklearn.model_selection import train_test_split from sklearn.datasets import load_breast_cancer import kerastuner as kt Step 2: Load and Preprocess Data # We’ll use the Breast Cancer dataset from sklearn, which is relatively small and great for this task.\n# Load the Breast Cancer dataset data = load_breast_cancer() X = data.data y = data.target # Split the dataset into training and validation sets X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) Breast Cancer Dataset: This dataset is used for binary classification (predicting whether cancer is benign or malignant). Train/Test Split: We split the data into training and validation sets (80%-20%). Step 3: Define the Hypermodel Function # We will create a function that uses Keras Tuner to explore different combinations of hyperparameters.\ndef build_model(hp): model = Sequential() # Tune the number of units in the first Dense layer hp_units = hp.Int(\u0026#39;units\u0026#39;, min_value=8, max_value=128, step=8) model.add(Dense(units=hp_units, activation=\u0026#39;relu\u0026#39;, input_shape=(X_train.shape[1],))) # Tune the learning rate for the optimizer hp_learning_rate = hp.Choice(\u0026#39;learning_rate\u0026#39;, values=[1e-2, 1e-3, 1e-4]) model.add(Dense(1, activation=\u0026#39;sigmoid\u0026#39;)) model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=hp_learning_rate), loss=\u0026#39;binary_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) return model hp.Int('units', min_value=8, max_value=128, step=8): This allows Keras Tuner to try different numbers of neurons in the first Dense layer, ranging from 8 to 128, in steps of 8. hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4]): This allows the tuner to try different learning rates. The model uses a Dense layer with a sigmoid activation for binary classification. Step 4: Set Up the Keras Tuner # We will use the RandomSearch tuner to find the best set of hyperparameters.\n# Set up the tuner tuner = kt.RandomSearch( build_model, objective=\u0026#39;val_accuracy\u0026#39;, max_trials=5, # Number of different models to try executions_per_trial=3, # Number of times to train each model directory=\u0026#39;my_dir\u0026#39;, project_name=\u0026#39;intro_to_kt\u0026#39; ) # Search for the best hyperparameters tuner.search(X_train, y_train, epochs=10, validation_data=(X_val, y_val)) kt.RandomSearch(): This tuner will randomly sample combinations of hyperparameters to find the best set. objective='val_accuracy': The tuner will try to maximize validation accuracy. max_trials=5: The tuner will test 5 different combinations of hyperparameters. executions_per_trial=3: Each model will be trained 3 times to get an average result, which adds stability to the evaluation. directory: This is the folder where Keras Tuner will store all the tuning logs and results. project_name: This helps organize different projects within the same directory. Step 5: Get the Best Model # Once the search is complete, we can get the best model and hyperparameters.\n# Get the best hyperparameters best_hps = tuner.get_best_hyperparameters(num_trials=1)[0] print(f\u0026#34;Best number of units: {best_hps.get(\u0026#39;units\u0026#39;)}\u0026#34;) print(f\u0026#34;Best learning rate: {best_hps.get(\u0026#39;learning_rate\u0026#39;)}\u0026#34;) # Build the model with the best hyperparameters best_model = tuner.hypermodel.build(best_hps) # Train the best model on the full training set history = best_model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=20) tuner.get_best_hyperparameters(): This retrieves the best set of hyperparameters found during the search. tuner.hypermodel.build(best_hps): Builds the best model based on these hyperparameters. Train the Best Model: We retrain the best model for a longer time (20 epochs) to see how well it performs. Step 6: Visualize Training Performance # We can visualize the training history to see how the model performs over time.\nimport matplotlib.pyplot as plt # Convert the history to a DataFrame for easy visualization import pandas as pd history_df = pd.DataFrame(history.history) # Plot training and validation accuracy plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;accuracy\u0026#39;], label=\u0026#39;Training Accuracy\u0026#39;) plt.plot(history_df[\u0026#39;val_accuracy\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Accuracy\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy\u0026#39;) plt.legend() plt.show() # Plot training and validation loss plt.figure(figsize=(10, 6)) plt.plot(history_df[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss\u0026#39;) plt.plot(history_df[\u0026#39;val_loss\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=\u0026#39;Validation Loss\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss\u0026#39;) plt.legend() plt.show() Training and Validation Accuracy/Loss: Plot the accuracy and loss to see how the model performed during training and if it generalized well to the validation set.\nVideo # ","date":"7 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-7/","section":"Challenges","summary":"Today marks Day 7 of my 30 Days, 30 Deep Learning Projects Challenge. The task for today is to Fine-tune hyperparameters with Keras Tuner on a small NN. Curious about how it went? Read on to see the results!","title":"Day 7: Fine-tune hyperparameters with Keras Tuner on a small NN","type":"challenge"},{"content":"Today, we\u0026rsquo;ll explore two key techniques for controlling overfitting: Dropout and L2 Regularization. We\u0026rsquo;ll apply these techniques to a model trained on the Titanic dataset, which is a well-known dataset containing information about passengers, like their age, gender, ticket class, etc., with the goal of predicting survival.\nWhat is Overfitting? # Overfitting occurs when a model learns the training data too well, including its noise and irrelevant details, and fails to generalize to new, unseen data. It’s like a student who memorizes every page of a book instead of understanding the main concepts — they might do well in a test with the exact same questions but will struggle when the questions are different. Two techniques to prevent overfitting are Dropout and L2 Regularization.\nWhat is Dropout? # Dropout is a technique where, during training, we randomly turn off some neurons in the network. In simple terms, dropout forces the network to learn more robust features instead of relying on just a few neurons to make predictions. What is L2 Regularization? # L2 Regularization is a technique where we penalize large weights in the neural network. The idea is to keep the weights small so that the model doesn’t rely too heavily on any particular feature. Plan: # Load the Titanic dataset and preprocess it. Build a neural network and apply Dropout and L2 Regularization. Train the model and observe the effect on overfitting. Step-by-Step Implementation # Step 1: Import Libraries and Load Data # First, we\u0026rsquo;ll import the necessary libraries and load the Titanic dataset using Pandas.\nimport pandas as pd import numpy as np import tensorflow as tf from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Dropout from tensorflow.keras.regularizers import l2 # Load Titanic dataset from seaborn (for simplicity) import seaborn as sns titanic = sns.load_dataset(\u0026#39;titanic\u0026#39;) # Display the first few rows to understand the structure print(titanic.head()) We\u0026rsquo;re using Seaborn to load the Titanic dataset for simplicity. You could use Pandas to load the dataset from a CSV as well. Display the first few rows to get an understanding of the data structure. Step 2: Preprocess the Data # We\u0026rsquo;ll need to preprocess the dataset:\nRemove unnecessary columns. Fill missing values. Convert categorical columns to numerical. # Drop columns that are not useful for prediction titanic = titanic.drop([\u0026#39;embarked\u0026#39;, \u0026#39;class\u0026#39;, \u0026#39;who\u0026#39;, \u0026#39;adult_male\u0026#39;, \u0026#39;alive\u0026#39;, \u0026#39;deck\u0026#39;, \u0026#39;embark_town\u0026#39;, \u0026#39;sex\u0026#39;], axis=1) # Fill missing values titanic[\u0026#39;age\u0026#39;].fillna(titanic[\u0026#39;age\u0026#39;].mean(), inplace=True) titanic[\u0026#39;fare\u0026#39;].fillna(titanic[\u0026#39;fare\u0026#39;].mean(), inplace=True) # Convert categorical columns to numerical titanic[\u0026#39;embarked\u0026#39;] = titanic[\u0026#39;embarked\u0026#39;].astype(\u0026#39;category\u0026#39;).cat.codes titanic[\u0026#39;sex\u0026#39;] = titanic[\u0026#39;sex\u0026#39;].astype(\u0026#39;category\u0026#39;).cat.codes # Split the dataset into features and labels X = titanic.drop(\u0026#39;survived\u0026#39;, axis=1) y = titanic[\u0026#39;survived\u0026#39;] # Split into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Standardize the data scaler = StandardScaler() X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test) Drop unnecessary columns: Columns like who, embark_town, sex are either redundant or not useful for training. Fill missing values: Filling missing values in age and fare helps maintain data consistency. Convert categorical columns: Convert categorical features (like sex) into numerical codes. Standardize the data: Scaling the features helps in faster and more efficient training. Step 3: Build the Model with Dropout and L2 Regularization # Next, we’ll build a neural network model with L2 Regularization and Dropout layers.\n# Build the neural network model with Dropout and L2 Regularization model = Sequential() model.add(Dense(128, activation=\u0026#39;relu\u0026#39;, input_shape=(X_train.shape[1],), kernel_regularizer=l2(0.01))) model.add(Dropout(0.5)) # Dropout rate of 50% model.add(Dense(64, activation=\u0026#39;relu\u0026#39;, kernel_regularizer=l2(0.01))) model.add(Dropout(0.3)) # Dropout rate of 30% model.add(Dense(1, activation=\u0026#39;sigmoid\u0026#39;)) # Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;binary_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) # Display the model summary model.summary() Dense(128, activation=\u0026lsquo;relu\u0026rsquo;, kernel_regularizer=l2(0.01)): Adds a hidden layer with 128 neurons and L2 regularization. The L2 penalty (0.01) helps keep weights from growing too large. Dropout(0.5): Randomly drops 50% of neurons in this layer during training. This prevents the model from being overly reliant on any specific neurons. The output layer is a single neuron with a sigmoid activation function, used for binary classification. Step 4: Train the Model # Now, we will train the model and observe how it performs over the epochs.\n# Train the model history = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=50, batch_size=32, verbose=1) # Convert the history to a DataFrame for easy viewing history_df = pd.DataFrame(history.history) # Display the first few rows of the training history DataFrame print(history_df.head()) model.fit(): Trains the model on the training data for 50 epochs with a batch size of 32. We convert the training history to a Pandas DataFrame to make it easy to visualize the training and validation performance. Step 5: Visualize Training and Validation Performance # We will plot the training and validation loss and accuracy to see how well the model is controlling overfitting.\nimport matplotlib.pyplot as plt # Plot training and validation loss plt.figure(figsize=(14, 6)) plt.plot(history_df[\u0026#39;loss\u0026#39;], label=\u0026#39;Training Loss\u0026#39;) plt.plot(history_df[\u0026#39;val_loss\u0026#39;], label=\u0026#39;Validation Loss\u0026#39;, linestyle=\u0026#39;--\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss\u0026#39;) plt.legend() plt.show() # Plot training and validation accuracy plt.figure(figsize=(14, 6)) plt.plot(history_df[\u0026#39;accuracy\u0026#39;], label=\u0026#39;Training Accuracy\u0026#39;) plt.plot(history_df[\u0026#39;val_accuracy\u0026#39;], label=\u0026#39;Validation Accuracy\u0026#39;, linestyle=\u0026#39;--\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy\u0026#39;) plt.legend() plt.show() Training vs Validation Loss: Look for differences between the training and validation loss. A significant gap could indicate overfitting. Training vs Validation Accuracy: A good model will have training and validation accuracies that are close to each other, showing that it generalizes well. Video # ","date":"6 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-6/","section":"Challenges","summary":"Today marks Day 6 of my 30 Days, 30 Deep Learning Projects Challenge. The task for today is to Apply dropout and regularization (L2) for overfitting control (Titanic Dataset). Curious about how it went? Read on to see the results!","title":"Day 6: Apply dropout and regularization (L2) for overfitting control (Titanic Dataset)","type":"challenge"},{"content":"Exploring optimizers is crucial because they control how a neural network updates its weights to minimize loss and find the best solution. For today\u0026rsquo;s task, we\u0026rsquo;ll use a pre-built Convolutional Neural Network (CNN) on the CIFAR-10 dataset and compare the impact of three popular optimizers: SGD, Adam, and RMSprop.\nOverview of the Problem # The CIFAR-10 dataset contains 60,000 color images across 10 categories (like airplanes, cars, birds, etc.). You will use a pre-built Convolutional Neural Network (CNN) architecture to train on this dataset. We\u0026rsquo;ll experiment with three optimizers: SGD (Stochastic Gradient Descent) Adam (Adaptive Moment Estimation) RMSprop (Root Mean Square Propagation) Brief Overview of the Optimizers # Optimizers are like guides that help your neural network find the best solution. Imagine your neural network is a hiker trying to find the lowest point in a hilly landscape (representing the minimum loss). The optimizer is the strategy or tool the hiker uses to get to the lowest point as quickly and efficiently as possible.\nRead here to know more about the optimizers.\nPlan for Comparison # Load and preprocess the CIFAR-10 dataset. Build a CNN model. Train the model using three different optimizers: SGD, Adam, and RMSprop. Compare the performance of each optimizer. Let’s get started!\nStep-by-Step Implementation # Step 1: Import Libraries and Load Data # import tensorflow as tf from tensorflow.keras.datasets import cifar10 from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense import matplotlib.pyplot as plt # Load CIFAR-10 dataset (X_train, y_train), (X_test, y_test) = cifar10.load_data() # Normalize pixel values to between 0 and 1 X_train = X_train / 255.0 X_test = X_test / 255.0 CIFAR-10 is a dataset of 60,000 color images (each 32x32 pixels) in 10 classes. Normalization of the pixel values between 0 and 1 helps improve the efficiency and stability of the training process. Step 2: Define a Function to Build a Simple CNN Model # def build_cnn(): model = Sequential() model.add(Conv2D(32, (3, 3), activation=\u0026#39;relu\u0026#39;, input_shape=(32, 32, 3))) model.add(MaxPooling2D((2, 2))) model.add(Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;)) model.add(MaxPooling2D((2, 2))) model.add(Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;)) model.add(Flatten()) model.add(Dense(64, activation=\u0026#39;relu\u0026#39;)) model.add(Dense(10, activation=\u0026#39;softmax\u0026#39;)) return model We define a function build_cnn() that returns a simple CNN model. Conv2D layers with ReLU activation extract features from images. MaxPooling2D reduces the dimensions of the feature maps. The final layers are fully connected (Dense) layers, ending with softmax to output 10 classes. Step 3: Train the Model with Different Optimizers # We will train the model using SGD, Adam, and RMSprop and compare their performance.\n# List of optimizers to compare optimizers = { \u0026#39;SGD\u0026#39;: tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.9), \u0026#39;Adam\u0026#39;: tf.keras.optimizers.Adam(learning_rate=0.001), \u0026#39;RMSprop\u0026#39;: tf.keras.optimizers.RMSprop(learning_rate=0.001) } # Dictionary to store the training history of each optimizer history_dict = {} for opt_name, opt in optimizers.items(): # Build and compile the model with a specific optimizer model = build_cnn() model.compile(optimizer=opt, loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) # Train the model print(f\u0026#34;Training with optimizer: {opt_name}\u0026#34;) history = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=64, verbose=1) # Store the history for comparison history_dict[opt_name] = history.history Optimizers:\nSGD(learning_rate=0.01, momentum=0.9): Includes momentum to make the updates smoother and potentially faster. Adam(learning_rate=0.001): Uses adaptive learning rates, popular for most use-cases. RMSprop(learning_rate=0.001): Also uses adaptive learning rates to stabilize training. Model Compilation and Training:\nEach model is compiled with a specific optimizer. epochs=10: We train each model for 10 epochs to get an initial comparison. Step 4: Visualize and Compare the Results # We will plot the training and validation accuracy and loss for each optimizer to compare their effectiveness.\n# Plot training and validation loss for each optimizer plt.figure(figsize=(14, 8)) for opt_name in optimizers.keys(): plt.plot(history_dict[opt_name][\u0026#39;loss\u0026#39;], label=f\u0026#39;Training Loss ({opt_name})\u0026#39;) plt.plot(history_dict[opt_name][\u0026#39;val_loss\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=f\u0026#39;Validation Loss ({opt_name})\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss for Different Optimizers\u0026#39;) plt.legend() plt.show() # Plot training and validation accuracy for each optimizer plt.figure(figsize=(14, 8)) for opt_name in optimizers.keys(): plt.plot(history_dict[opt_name][\u0026#39;accuracy\u0026#39;], label=f\u0026#39;Training Accuracy ({opt_name})\u0026#39;) plt.plot(history_dict[opt_name][\u0026#39;val_accuracy\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=f\u0026#39;Validation Accuracy ({opt_name})\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy for Different Optimizers\u0026#39;) plt.legend() plt.show() Training vs Validation Loss and Accuracy: We plot both the training and validation performance for each optimizer. This will help you visually understand how well each optimizer performs in terms of both learning speed and generalization. What to Look for in the Results? # Training and Validation Loss\nLower loss is better. Look for the speed at which the loss decreases. Faster convergence indicates that the model is learning well. Validation loss should be similar to training loss. If validation loss is much higher, it suggests overfitting. Training and Validation Accuracy\nHigher accuracy is better. Check how quickly the model reaches a high accuracy and whether it generalizes well to the test set. Expected Observations # SGD:\nTraining Loss and Accuracy: May take longer to converge, and the loss might not decrease as steadily. Validation Performance: Often less stable than adaptive optimizers, but with enough epochs, it can catch up. Adam:\nTraining Loss and Accuracy: Expected to have faster convergence and stable learning. You should see the loss decreasing quickly, and accuracy improving steadily. Validation Performance: Usually good, with high validation accuracy that is close to training accuracy, suggesting good generalization. RMSprop:\nTraining Loss and Accuracy: Similar to Adam, RMSprop often works well with adaptive learning rates and can converge quickly. Validation Performance: Typically shows a steady improvement, often comparable to Adam. Video # ","date":"5 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-5/","section":"Challenges","summary":"Today marks Day 5 of my 30 Days, 30 Deep Learning Projects Challenge. The task for today is to Experiment with optimizers (SGD, Adam, RMSprop) using a pre-built CNN on CIFAR-10. Curious about how it went? Read on to see the results!","title":"Day 5: Experiment with optimizers (SGD, Adam, RMSprop) using a pre-built CNN on CIFAR-10","type":"challenge"},{"content":"Comparing activation functions is key to understanding how they influence a neural network\u0026rsquo;s ability to learn and generalize. Today, we\u0026rsquo;ll explore the Fashion MNIST dataset and compare the effect of different activation functions: ReLU, Sigmoid, and Tanh on the model\u0026rsquo;s performance.\nProblem Outline # We’ll train multiple neural network models on the Fashion MNIST dataset. We will vary the activation functions used in the hidden layers: ReLU, Sigmoid, and Tanh. We will observe how each activation function affects training speed, accuracy, and generalization. What Are Activation Functions? # ReLU (Rectified Linear Unit)\nReLU stands for Rectified Linear Unit. Think of it as a function that passes positive values as they are, and stops negative values (turns them into zero). Sigmoid\nThe Sigmoid function squashes input values into a range between 0 and 1. It’s useful when you want to decide something in a probabilistic way, for example, \u0026ldquo;Is this true or not?\u0026rdquo; because 0 and 1 represent extremes. Tanh\nTanh stands for Hyperbolic Tangent. It’s very similar to Sigmoid, but instead of outputting values between 0 and 1, it outputs between -1 and 1. This is helpful because it’s centered around zero, which makes learning easier for some neural networks. Read here for an easy explaination with examples (without complex math).\nPlan for Comparison # Load and preprocess the Fashion MNIST dataset. Create three identical neural network architectures, changing only the activation function. Train each model and compare their performance in terms of accuracy, loss, and training speed. Step-by-Step Implementation # Step 1: Import Libraries and Load Data # import numpy as np import tensorflow as tf from tensorflow.keras.datasets import fashion_mnist from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Flatten, Dense import matplotlib.pyplot as plt # Load the Fashion MNIST dataset (X_train, y_train), (X_test, y_test) = fashion_mnist.load_data() # Normalize the data to the range [0, 1] X_train = X_train / 255.0 X_test = X_test / 255.0 Fashion MNIST is a dataset of grayscale images of different types of clothing. We normalize the pixel values to the range [0, 1] to make training more efficient. Step 2: Define a Function to Build the Model with Different Activation Functions # def build_model(activation_function): model = Sequential() model.add(Flatten(input_shape=(28, 28))) # Flatten the images to a vector model.add(Dense(128, activation=activation_function)) # First hidden layer model.add(Dense(64, activation=activation_function)) # Second hidden layer model.add(Dense(10, activation=\u0026#39;softmax\u0026#39;)) # Output layer for 10 classes return model build_model(activation_function): A function that takes the activation function as input and returns a model using that activation function. We use two hidden layers with 128 and 64 neurons respectively, and an output layer with 10 neurons (one for each clothing category). Step 3: Compile and Train Models with Different Activation Functions # We’ll compare ReLU, Sigmoid, and Tanh by training the same model architecture with each.\n# List of activation functions to compare activation_functions = [\u0026#39;relu\u0026#39;, \u0026#39;sigmoid\u0026#39;, \u0026#39;tanh\u0026#39;] # Dictionary to store the training history of each model history_dict = {} for activation in activation_functions: # Build and compile the model model = build_model(activation_function=activation) model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) # Train the model print(f\u0026#34;Training with activation function: {activation}\u0026#34;) history = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=32, verbose=1) # Store the history for comparison history_dict[activation] = history.history activation_functions: The list of activation functions we want to compare. model.compile(): We use the Adam optimizer. loss='sparse_categorical_crossentropy' is used since we have multiple classes. history_dict[activation] saves the training history for each activation function to compare later. Step 4: Visualize and Compare the Results # We’ll plot the training and validation accuracy as well as the loss for each activation function to compare them.\n# Plot training and validation loss for each activation function plt.figure(figsize=(14, 8)) for activation in activation_functions: plt.plot(history_dict[activation][\u0026#39;loss\u0026#39;], label=f\u0026#39;Training Loss ({activation})\u0026#39;) plt.plot(history_dict[activation][\u0026#39;val_loss\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=f\u0026#39;Validation Loss ({activation})\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss for Different Activation Functions\u0026#39;) plt.legend() plt.show() # Plot training and validation accuracy for each activation function plt.figure(figsize=(14, 8)) for activation in activation_functions: plt.plot(history_dict[activation][\u0026#39;accuracy\u0026#39;], label=f\u0026#39;Training Accuracy ({activation})\u0026#39;) plt.plot(history_dict[activation][\u0026#39;val_accuracy\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=f\u0026#39;Validation Accuracy ({activation})\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy for Different Activation Functions\u0026#39;) plt.legend() plt.show() Training vs Validation Loss and Accuracy: We plot both the training and validation performance for each activation function to visually compare how well each one learns and generalizes.\nWhat to Look for in the Results? # Training and Validation Loss\nLower loss is better. Look for the speed at which the loss decreases. Faster convergence indicates that the model is learning well. Validation loss should be similar to training loss. If validation loss is much higher, it suggests overfitting. Training and Validation Accuracy\nHigher accuracy is better. Check how quickly the model reaches a high accuracy and whether it generalizes well to the test set. Expected Observations: # ReLU: # Training Loss and Accuracy: ReLU is often the best for deeper networks because it doesn’t suffer from the vanishing gradient problem. It tends to converge faster and often achieves higher accuracy compared to other activations. Validation Performance: Usually matches training performance well, indicating good generalization. Sigmoid: # Training Loss and Accuracy: Sigmoid can suffer from the vanishing gradient problem, especially when there are multiple layers. This means it could learn more slowly and not reach as high an accuracy within a few epochs. Validation Performance: Sigmoid can sometimes overfit due to saturation, leading to poorer performance on validation data compared to ReLU or Tanh. Tanh: # Training Loss and Accuracy: Tanh works similarly to Sigmoid but is centered at zero, which often allows better learning dynamics because the average output is closer to zero, reducing bias in subsequent layers. Validation Performance: Often has better performance compared to Sigmoid but usually slower than ReLU. Video # Coming Soon. ","date":"4 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-4/","section":"Challenges","summary":"Today marks Day 4 of my 30 Days, 30 Deep Learning Projects Challenge. The task for today is to Compare activation functions (ReLU, Sigmoid, Tanh) on Fashion MNIST. Curious about how it went? Read on to see the results!","title":"Day 4: Compare activation functions (ReLU, Sigmoid, Tanh) on Fashion MNIST","type":"challenge"},{"content":" What is Backpropagation? # Backpropagation is the algorithm that makes neural networks learn by adjusting their weights. Essentially, it\u0026rsquo;s the learning process.\nIt works by calculating the error in predictions (the difference between predicted and true values) and then propagating this error backward through the network to update the weights in each layer so that the model can make better predictions in the future.\nHow Backpropagation Works in Simple Terms:\nForward Pass: The input data is passed through the network layer by layer to produce an output. Calculate Loss: The output is compared to the actual labels to calculate the loss (the error). Backward Pass (Gradient Calculation): The loss is used to compute the gradient of the error with respect to each weight. Weight Update: The weights are updated in the direction that minimizes the loss using the gradient and a learning rate. Learning Rate: What Does It Do?\nThe learning rate controls how big of a step we take in the direction of minimizing the error. High Learning Rate: Faster learning but might overshoot the minimum and miss convergence. Low Learning Rate: More stable, but can be slow and get stuck in local minima. Today, we\u0026rsquo;ll experiment with different learning rates to see their effect on training.\nLet\u0026rsquo;s Explore Practically with MNIST # We\u0026rsquo;ll modify our MNIST classifier and test different learning rates to observe the impact. Here\u0026rsquo;s how we’ll proceed:\nLoad the MNIST Dataset. Build the neural network. Compile the model with different learning rates. Train the model and observe how changing learning rates impacts the training process. Plot the training and validation accuracy/loss to visualize the effect of different learning rates. Here’s the full code along with detailed explanations:\nStep-by-Step Implementation # Step 1: Load Libraries and the MNIST Dataset # import numpy as np import pandas as pd import tensorflow as tf from tensorflow.keras.datasets import mnist from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Flatten import matplotlib.pyplot as plt # Load the MNIST dataset (X_train, y_train), (X_test, y_test) = mnist.load_data() # Normalize the data to range 0-1 X_train = X_train / 255.0 X_test = X_test / 255.0 Libraries:\nnumpy: Used for numerical operations. tensorflow: We’ll use Keras, which is a part of TensorFlow, to create our neural network. mnist: The MNIST dataset is built into Keras, which makes it easy to load. Sequential and Dense: These help in building a neural network. Sequential is used for stacking layers. matplotlib: Used for visualizing the digits from the dataset. We load the data and then normalize the pixel values between 0 and 1 to improve the training process.\nStep 2: Build the Neural Network Model # # Build a simple feedforward neural network model model = Sequential() model.add(Flatten(input_shape=(28, 28))) # Flatten the 28x28 images to 784-length vectors model.add(Dense(128, activation=\u0026#39;relu\u0026#39;)) # First hidden layer with 128 neurons model.add(Dense(64, activation=\u0026#39;relu\u0026#39;)) # Second hidden layer with 64 neurons model.add(Dense(10, activation=\u0026#39;softmax\u0026#39;)) # Output layer with 10 neurons (for digits 0-9) We use a Flatten layer to convert the 28x28 image into a 1D vector. Then, we use Dense layers with ReLU activation for learning complex patterns. The output layer uses softmax to produce a probability distribution over 10 classes (digits 0-9). Step 3: Experiment with Different Learning Rates # Now we’ll create multiple versions of the same model but with different learning rates:\n# Different learning rates to experiment with learning_rates = [0.01, 0.001, 0.0001] # Store training history for each learning rate history_dict = {} for lr in learning_rates: # Compile the model with a specific learning rate model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=lr), loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) # Train the model print(f\u0026#34;Training with learning rate: {lr}\u0026#34;) history = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=32, verbose=1) # Save the history for plotting history_dict[lr] = history.history Learning Rates: We’re testing with three different values: 0.01: High learning rate. 0.001: Standard default learning rate. 0.0001: Low learning rate Loop through Learning Rates: We compile and train the model for each learning rate. optimizer=tf.keras.optimizers.Adam(learning_rate=lr) allows us to use the Adam optimizer with a specific learning rate. history_dict[lr] saves the training history for later comparison. Step 4: Visualize the Effect of Different Learning Rates # After training, we can visualize the training and validation accuracy and loss for different learning rates to see their impact:\n# Plot training and validation loss for different learning rates plt.figure(figsize=(14, 8)) for lr in learning_rates: plt.plot(history_dict[lr][\u0026#39;loss\u0026#39;], label=f\u0026#39;Training Loss (lr={lr})\u0026#39;) plt.plot(history_dict[lr][\u0026#39;val_loss\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=f\u0026#39;Validation Loss (lr={lr})\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation Loss for Different Learning Rates\u0026#39;) plt.legend() plt.show() # Plot training and validation accuracy for different learning rates plt.figure(figsize=(14, 8)) for lr in learning_rates: plt.plot(history_dict[lr][\u0026#39;accuracy\u0026#39;], label=f\u0026#39;Training Accuracy (lr={lr})\u0026#39;) plt.plot(history_dict[lr][\u0026#39;val_accuracy\u0026#39;], linestyle=\u0026#39;--\u0026#39;, label=f\u0026#39;Validation Accuracy (lr={lr})\u0026#39;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy for Different Learning Rates\u0026#39;) plt.legend() plt.show() We plot the training and validation loss as well as accuracy for each learning rate. This will help us visually understand how different learning rates affect the training process. Observations # Learning Rate Effects: # High Learning Rate (0.01): Likely unstable, as it makes large steps towards the minimum of the loss function, potentially overshooting. You may notice fluctuating or diverging losses and unstable accuracy. Default Learning Rate (0.001): Often results in a good balance between fast convergence and stability. Training and validation loss will gradually decrease, and accuracy will steadily improve. Low Learning Rate (0.0001): Training will be much slower, and the model might take a lot of time to converge. Loss might decrease very gradually, and you might see the accuracy improving slowly, which could lead to longer training times. Video # Coming Soon. ","date":"3 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-3/","section":"Challenges","summary":"Today marks Day 3 of my 30 Days, 30 Deep Learning Projects Challenge. The task for today is to Explore backpropagation theory and tweak learning rates (MNIST). Curious about how it went? Read on to see the results!","title":"Day 3: Explore backpropagation theory and tweak learning rates (MNIST)","type":"challenge"},{"content":" Activation Functions: # 1. ReLU (Rectified Linear Unit) # What is ReLU? # ReLU stands for Rectified Linear Unit. Think of it as a function that passes positive values as they are, and stops negative values (turns them into zero). Simple Example - Turning Off the Negative Signal:\nImagine you’re listening to a music player. The volume knob can go from -10 to +10, where negative values mean turning the volume down (even making it mute) and positive values mean turning it up. However, ReLU doesn’t care about the negative — it just ignores all negative values and keeps positive values as they are.\nIf the volume knob is at +7, ReLU will let 7 through. If the volume knob is at -5, ReLU will say, \u0026ldquo;Nope, it’s negative,\u0026rdquo; and turn it into 0 (muted). In Neural Networks: # ReLU acts as a gate. If the input is positive, ReLU lets it pass; if it’s negative, it blocks it and turns it into 0. This helps the network learn quickly because it only needs to deal with positive signals or zeros, and it avoids problems caused by too many small negative values.\nSummary of ReLU:\nPositive input: Let it through as-is. Negative input: Turn it into 0. Analogy: It’s like a music volume knob that ignores negative adjustments and lets positive adjustments through. 2. Sigmoid # What is Sigmoid? # The Sigmoid function squashes input values into a range between 0 and 1. It’s useful when you want to decide something in a probabilistic way, for example, \u0026ldquo;Is this true or not?\u0026rdquo; because 0 and 1 represent extremes. Simple Example - Light Dimmer:\nImagine a light dimmer in a room. You have a knob to control the brightness of the light, but the light can only be between off (0) and full brightness (1).\nIf you turn the knob slightly, you get a value close to 0 (dim light). If you turn it more, you get something close to 1 (bright light). If the knob is in the middle, you get a partial value, like 0.5 (medium light). In Neural Networks:\nSigmoid takes the input, and squeezes it between 0 and 1. It’s often used in the output layer for binary classification, where you need to determine if something is one category or another (e.g., cat or no cat).\nSummary of Sigmoid:\nInput range: Squashes everything between 0 and 1. Use case: When you want to output something in the form of probabilities. Analogy: It’s like a light dimmer that smoothly adjusts brightness between off (0) and fully on (1). 3. Tanh (Hyperbolic Tangent) # What is Tanh? # Tanh stands for Hyperbolic Tangent. It’s very similar to Sigmoid, but instead of outputting values between 0 and 1, it outputs between -1 and 1. This is helpful because it’s centered around zero, which makes learning easier for some neural networks. Simple Example - Thermometer:\nImagine you have a thermometer that can measure both cold and hot temperatures, with the following scale:\nNegative values mean cold temperatures. Positive values mean hot temperatures. Zero means the temperature is neutral (neither hot nor cold). If it’s cold, the thermometer shows a negative value like -0.7. If it’s hot, it shows a positive value like +0.8. If it’s a neutral temperature, it shows 0. In Neural Networks:\nTanh lets values be both positive or negative. If a neuron wants to strongly say \u0026ldquo;this input is important in a positive way,\u0026rdquo; it outputs a high positive value (close to +1). If it wants to say, \u0026ldquo;this input has a negative effect,\u0026rdquo; it outputs a value close to -1. This means Tanh is good for situations where being able to indicate both strong positive and negative signals is useful. Summary of Tanh:\nOutput range: Values between -1 and 1. Use case: Useful for representing signals that can be either positive or negative, centered around zero. Analogy: It’s like a thermometer showing values between cold (negative), neutral (zero), and hot (positive). What are Optimizers? # Optimizers are like guides that help your neural network find the best solution. Imagine your neural network is a hiker trying to find the lowest point in a hilly landscape (representing the minimum loss). The optimizer is the strategy or tool the hiker uses to get to the lowest point as quickly and efficiently as possible.\nOptimizer 1: SGD (Stochastic Gradient Descent) # Simple Analogy: A Hiker Taking Small Steps Down a Hill\nImagine you\u0026rsquo;re a hiker trying to reach the lowest point in a valley. You can only see the part of the hill right around you, and you take steps downward in the direction that looks like it leads lower. With SGD, each step is like taking a small, careful move down the hill based on the current slope you feel under your feet. Characteristics of SGD:\nCareful Steps: You only take steps based on local information. If the ground feels like it\u0026rsquo;s sloping down, you take a step that way. Slow Progress: Because you’re taking small steps and sometimes relying only on what’s directly around you, you may not always find the fastest path down. Momentum: You can improve SGD by adding momentum, which is like letting the hiker pick up some speed when going downhill. This helps you avoid getting stuck on small bumps and can lead you more directly to the lowest point. In a Neural Network:\nSGD works by adjusting each parameter (weight) a little bit at a time in the direction that reduces error. It’s straightforward but can be slow, especially when dealing with complex landscapes where there are lots of ups and downs (e.g., in deep networks). Optimizer 2: Adam (Adaptive Moment Estimation) # Simple Analogy: A Hiker with Adaptive Gear and Memory\nImagine the same hiker again, but this time you’re equipped with some special gear: You have adaptive boots that can adjust based on how steep the terrain is. You also have a notebook to remember the directions you’ve tried before. Characteristics of Adam:\nAdaptive Steps: The boots can adjust their step size based on how steep the slope is. If it’s a gentle slope, they take larger steps. If it’s steep, they take smaller, careful steps. Memory: The notebook helps you remember the directions you\u0026rsquo;ve tried before. If you notice that you’ve been heading downhill steadily, you keep going in that direction with more confidence. In a Neural Network:\nAdam combines the advantages of momentum (like remembering where you’re going) and adaptive learning rates (like adjusting the step size). It’s a smart hiker that adapts based on the terrain, making it both fast and effective. That’s why Adam is often a favorite choice for training — it adjusts well, speeds up when it’s safe, and slows down when it needs to be careful. Optimizer 3: RMSprop (Root Mean Square Propagation) # Simple Analogy: A Hiker with Shock Absorbers to Smooth the Descent\nImagine the hiker again, but this time with shock-absorbing shoes. These shoes help you move steadily downhill without bouncing too much, even when the terrain gets a bit rocky. RMSprop is all about keeping the journey smooth and preventing big jumps that could lead you in the wrong direction. Characteristics of RMSprop:\nSmooth Descent: The shock absorbers help smooth out the steps so that you don’t make large jumps that could lead to getting stuck or accidentally going the wrong way. Short Memory: Unlike Adam, which remembers a lot, RMSprop focuses on the recent terrain and makes adjustments based on that. In a Neural Network:\nRMSprop adjusts the step size based on the recent history of the gradients. It’s especially useful when the terrain (i.e., the loss landscape) is bumpy, which often happens in deeper neural networks. It helps ensure that steps are not too big, preventing the network from missing the optimum point. When to Use Which Optimizer? # SGD:\nBest for simpler tasks or when you want more control over the learning process. Works well when combined with momentum. Good if you don’t need adaptive learning and have a good learning rate already. Adam:\nThe most popular choice for many deep learning problems. Works well for almost all use-cases, particularly when training deep networks. It’s fast and usually converges to a good solution with less tuning. RMSprop:\nSimilar to Adam, RMSprop is great for complex, deep networks where the terrain is uncertain. It’s a good choice when you need steady training without large jumps. Overfitting Techniques: # What is Overfitting? # Overfitting occurs when a model learns the training data too well, including its noise and irrelevant details, and fails to generalize to new, unseen data. It’s like a student who memorizes every page of a book instead of understanding the main concepts — they might do well in a test with the exact same questions but will struggle when the questions are different. Two techniques to prevent overfitting are Dropout and L2 Regularization.\n1. Dropout - An Easy Explanation # What is Dropout? # Dropout is a technique where, during training, we randomly turn off some neurons in the network. In simple terms, dropout forces the network to learn more robust features instead of relying on just a few neurons to make predictions. Analogy: A Team Project\nImagine you’re working on a team project with several teammates, and each person has a unique set of skills. If everyone knows that one teammate will always do the most important part (let\u0026rsquo;s say Sarah always takes the lead), people might start depending on her too much and not learn the other tasks well enough.\nBut suppose, during a practice run, you decide that Sarah won’t participate. This forces the rest of the team to step up and learn how to complete the project even without her.\nThe next time, someone else, maybe John, is not allowed to contribute, forcing everyone else to adjust and learn their part more thoroughly. The outcome?\nEvery member of the team is prepared and has developed a well-rounded set of skills. No one person is indispensable, and the project can be completed even if someone is unavailable. How Dropout Works: # In a neural network, dropout randomly turns off some neurons in each training iteration. The percentage of neurons that are turned off is called the dropout rate. If the rate is 50%, half of the neurons in that layer are turned off. By doing this, the model is forced to not rely on just a few neurons and instead distribute learning across many neurons. Effect of Dropout: # Better Generalization: Since the model can’t rely too much on specific neurons, it learns more generalized features, improving performance on new, unseen data. Prevents Overfitting: Dropout adds randomness and makes the training process less dependent on specific weights, preventing overfitting. Example in a Neural Network: # Imagine a layer of 8 neurons in a network:\n[ O O O O O O O O ] \u0026lt;- 8 neurons During dropout, if we apply a dropout rate of 50%, it randomly turns off 4 of these neurons:\n[ O X O X O O X X ] \u0026lt;- \u0026#39;X\u0026#39; means turned off Only the remaining neurons are active and contribute during that training iteration. In the next iteration, another set of neurons may be turned off randomly.\nIn Short: Dropout is like randomly telling team members they can’t participate in every practice, forcing everyone else to learn all parts of the project, making the entire team stronger.\n2. L2 Regularization - An Easy Explanation # What is L2 Regularization? # L2 Regularization is a technique where we penalize large weights in the neural network. The idea is to keep the weights small so that the model doesn’t rely too heavily on any particular feature. Analogy: The Minimalist Student\nImagine a student who is studying for a test. The student has a huge set of notes but decides that highlighting every single line will help them remember everything. This approach makes the notes bulky and hard to review effectively — and the student ends up memorizing unnecessary details.\nNow, imagine that someone tells the student that they’ll be penalized for highlighting too much. They need to limit themselves to highlighting only the most important parts of their notes.\nThe student now has to carefully decide what’s most important, and in the process, they learn the main concepts rather than trying to memorize everything. How L2 Regularization Works: # In a neural network, the goal is to find the best weights that minimize the loss. L2 Regularization adds a penalty to the loss function based on the magnitude of the weights. The idea is to keep weights small, making the network simpler and less likely to overfit. Mathematically (without going into too much detail): # The loss function is what the model tries to minimize. With L2 Regularization, a penalty term proportional to the sum of the squares of all weights is added to the loss. The larger the weights become, the larger the penalty. Effect of L2 Regularization: # Prevents Large Weights: L2 discourages weights from growing too large, which can lead to overfitting. Smoother Decision Boundary: With smaller weights, the decision boundaries formed by the model are generally simpler and smoother, which helps in generalizing to new data. Example in a Neural Network: # Suppose a neural network has learned that certain weights are very large because it thinks those features are extremely important. With L2 Regularization, the model tries to keep those weights smaller by adding a penalty to the loss function if weights grow too large. The result is that the network learns to spread importance across multiple features rather than relying heavily on just a few. In Short: L2 Regularization is like telling a student they can only highlight the most important information, preventing them from relying too much on unnecessary details, resulting in more efficient learning.\nVisual Example - Dropout vs L2 Regularization # Dropout: Imagine a soccer team where different players are randomly asked to sit out during practice games.\nResult: Every player improves overall skills because they can’t rely on specific key players during training. L2 Regularization: Imagine you are carrying a backpack up a mountain, and you have too many unnecessary items in it.\nYou get penalized (it becomes too hard to carry) if you try to bring everything, so you decide to bring only essential items. Result: You reach the top more efficiently because your backpack is lighter and more manageable. Both Dropout and L2 Regularization are widely used because they help a model generalize better by either simplifying the model or adding redundancy to the learning process, preventing over-reliance on specific pathways.\nWhat is a Convolutional Neural Network (CNN)? # A Convolutional Neural Network (CNN) is a type of deep learning model specially designed to work with images. CNNs can recognize patterns in images, much like how we use our eyes and brain to recognize faces, objects, and everything around us.\nThink of a CNN as a series of layers that each work together to identify features in an image, much like how our brain processes visual information step by step.\nImagine a Simple Example: Recognizing a Cat Picture # Imagine you’re looking at a picture of a cat. How do you know it’s a cat? Well, your brain processes the picture in parts. You might notice the whiskers, the eyes, the ears, and the shape of the face. Similarly, a CNN looks at the picture and breaks it down into parts to decide if it’s a cat.\nHow CNNs Work: Layers that Process the Image Step by Step # A CNN is composed of a series of layers, each working as a specialist to examine different parts of the image, each time getting more detailed. Here’s how it works:\nStep 1: The Input Image # An image is like a big grid of numbers. For a color image, each pixel is represented by three values (Red, Green, Blue, or RGB). Imagine an image as a large piece of graph paper filled with numbers, each number representing the brightness of a pixel.\nLayers of a CNN: Step-by-Step Analogy # Layer 1: Convolution Layer (Feature Detection) # Convolution is like scanning the image through a small window or filter. Imagine taking a small square magnifying glass and moving it over different parts of the image to look for specific details, like edges.\nReal-Life Analogy: Think of Convolution as a cookie-cutter: Imagine you’re trying to find if a cookie has chocolate chips. You use a small cookie-cutter tool to move around and find pieces of chocolate. This tool keeps moving over different parts of the cookie (image), and each time it detects a chocolate chip, it marks it. In CNNs, the filter (or \u0026ldquo;cookie-cutter\u0026rdquo;) moves over the entire image and creates a new version of the image, showing where specific features like edges or colors are located. Layer 2: Pooling Layer (Simplifying the Image) # Pooling simplifies information to make it easier to process. After detecting features, we want to make our image smaller while still keeping the important parts.\nReal-Life Analogy: Imagine you take a photo and then want to keep only the important details. Max Pooling is like using a tool to find the most important piece from each small section of the image. It reduces the size of the image and focuses on the prominent features. It’s like shrinking a picture but still being able to tell what’s in it — we keep the essential details. Layer 3: Flattening (Converting to a List) # After convolution and pooling, we end up with smaller images that hold key information. Flattening takes this shrunken version of the image and converts it into a list of numbers that can be fed to the final decision layer.\nReal-Life Analogy: Imagine you take all the essential pieces you found in the image and put them in a single row. This step turns all the detected features into a long list of important points. Layer 4: Fully Connected Layer (Making Decisions) # After flattening, the list of numbers is fed into the Fully Connected Layer, which acts as the decision maker. It looks at the entire list and decides what the image most likely is.\nReal-Life Analogy: Think of the Fully Connected Layer as your brain: After seeing whiskers, pointy ears, and a fluffy tail, your brain says: “Aha, this must be a cat!” The fully connected layer looks at all the details gathered and then combines them to identify the object. Putting It All Together: # Input Image: A picture of a cat is given to the CNN. Convolution Layer: The CNN applies filters to detect basic features, like edges or colors. Pooling Layer: The network simplifies the image, keeping only the essential information. Flattening: It turns the pooled image into a list of key features. Fully Connected Layer: The CNN examines the list of features and decides if it’s a cat, dog, bird, or something else. Why Do CNNs Work Well for Images? # Local Patterns: Images have local patterns, like the arrangement of pixels in edges or textures. CNNs are excellent at detecting these patterns. Layer-by-Layer Detail: As you go deeper into a CNN, the network’s understanding becomes more detailed. Early layers recognize basic shapes (like lines), while deeper layers identify complex features (like eyes or fur patterns). Example of How CNNs Learn: # Imagine teaching a child to recognize animals in pictures:\nStep 1: Start by teaching them to look for simple shapes — maybe pointy ears, whiskers, and a tail. Step 2: Help them combine those shapes into more complex features, like recognizing that pointy ears and a fluffy body mean “cat.” Step 3: Finally, they learn to say, “That’s a cat!” by combining all the details. A CNN does something similar:\nEarly layers recognize simple features like edges. Middle layers detect patterns and textures. Later layers learn to identify complex objects like cats or cars. Recap with an Everyday Example: # Imagine looking at a puzzle with different parts that come together to reveal a picture:\nConvolutional layers look at each puzzle piece, identifying its shape and color. Pooling layers simplify it, keeping only the most important parts. Flattening and fully connected layers understand the entire puzzle and finally decide, “This puzzle is a picture of a cat.” Summary: # Convolutional Neural Networks (CNNs) are like pattern detectors that scan images in layers. Convolutional Layers detect basic features like edges and textures. Pooling Layers simplify images by reducing their size but keeping essential details. Fully Connected Layers make a decision based on detected features to identify the image. Visualization to Imagine: # Think of a camera with different filters:\nFirst, it uses an edge-detecting filter to find outlines. Then it simplifies the image, zooming out to keep key parts. Finally, it passes this “zoomed-out, highlighted version” to a decision-making brain, which says, “This is a cat.” ","date":"2 November 2024","externalUrl":null,"permalink":"/post/dl-faq/","section":"Post","summary":"This page contains an easy explaination of common jargons of Deep Learning.","title":"Deep Learning FAQ","type":"post"},{"content":"","date":"2 November 2024","externalUrl":null,"permalink":"/categories/dl/","section":"Categories","summary":"","title":"Dl","type":"categories"},{"content":"","date":"2 November 2024","externalUrl":null,"permalink":"/tags/dl/","section":"Tags","summary":"","title":"Dl","type":"tags"},{"content":"","date":"2 November 2024","externalUrl":null,"permalink":"/categories/ml/","section":"Categories","summary":"","title":"Ml","type":"categories"},{"content":"The MNIST dataset is one of the most popular datasets for learning the basics of machine learning and neural networks. It contains handwritten digits (0-9), and our goal is to build a neural network that can classify these digits.\nWe\u0026rsquo;ll use a Simple Neural Network (i.e., a Feedforward Neural Network) for this task. Let’s break down the solution into easy-to-follow steps, and I\u0026rsquo;ll explain every part so you understand what\u0026rsquo;s happening. We will use Keras, a high-level API in TensorFlow, to make this as simple as possible.\nStep-by-Step Solution Outline: # Load the MNIST Dataset. Prepare and Preprocess the Data. Build a Simple Feedforward Neural Network. Compile and Train the Model. Evaluate the Model Performance. Make Predictions (optional step to visualize some predictions). Here’s the full code along with detailed explanations:\nStep-by-Step Implementation # Step 1: Import Libraries and Load Data # import numpy as np import tensorflow as tf from tensorflow.keras.datasets import mnist from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Flatten import matplotlib.pyplot as plt numpy: Used for numerical operations. tensorflow: We’ll use Keras, which is a part of TensorFlow, to create our neural network. mnist: The MNIST dataset is built into Keras, which makes it easy to load. Sequential and Dense: These help in building a neural network. Sequential is used for stacking layers. matplotlib: Used for visualizing the digits from the dataset. Step 2: Load the MNIST Dataset # # Load the MNIST dataset (X_train, y_train), (X_test, y_test) = mnist.load_data() # Check the shape of the data print(f\u0026#34;Training data shape: {X_train.shape}, Training labels shape: {y_train.shape}\u0026#34;) print(f\u0026#34;Testing data shape: {X_test.shape}, Testing labels shape: {y_test.shape}\u0026#34;) mnist.load_data() loads the MNIST data and splits it into training and testing sets. X_train: Images of handwritten digits used for training the model. y_train: Corresponding labels for the training images (digits 0-9). X_test and y_test are the images and labels used for testing. Shapes: X_train.shape: The shape is (60000, 28, 28), which means we have 60,000 images, each with a size of 28x28 pixels. y_train.shape: We have 60,000 labels corresponding to the training images. Step 3: Preprocess the Data # # Normalize the data to range 0-1 X_train = X_train / 255.0 X_test = X_test / 255.0 # Flatten the images from 28x28 to 784 (since a dense layer expects a vector input) X_train = X_train.reshape(-1, 28 * 28) X_test = X_test.reshape(-1, 28 * 28) Normalize the Data: X_train / 255.0: The original pixel values are between 0 and 255. Dividing by 255 scales these values to between 0 and 1, which helps the neural network learn faster. Flatten the Images: The MNIST images are 28x28 pixels, which we need to flatten into a vector of 784 pixels (28 * 28). This is because a fully connected (Dense) layer expects 1D vectors rather than 2D images. Step 4: Build the Neural Network # # Build a simple feedforward neural network model model = Sequential() model.add(Dense(128, activation=\u0026#39;relu\u0026#39;, input_shape=(784,))) # First hidden layer with 128 neurons model.add(Dense(64, activation=\u0026#39;relu\u0026#39;)) # Second hidden layer with 64 neurons model.add(Dense(10, activation=\u0026#39;softmax\u0026#39;)) # Output layer with 10 neurons (for digits 0-9) Sequential(): We create a Sequential model to stack layers one after another. Dense(128, activation=\u0026lsquo;relu\u0026rsquo;, input_shape=(784,)): 128 neurons in the first hidden layer with ReLU activation function. input_shape=(784,) tells the model that the input will be a vector of length 784 (flattened image). Dense(64, activation=\u0026lsquo;relu\u0026rsquo;): Adds a second hidden layer with 64 neurons and ReLU activation. Dense(10, activation=\u0026lsquo;softmax\u0026rsquo;): The output layer has 10 neurons, each representing one of the digits 0-9. softmax activation is used to turn the output into probabilities, with the sum equal to 1. Step 5: Compile the Model # model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) optimizer=\u0026lsquo;adam\u0026rsquo;: The Adam optimizer is a popular choice for training deep learning models. loss=\u0026lsquo;sparse_categorical_crossentropy\u0026rsquo;: We use categorical cross-entropy since we have multiple classes (0-9) to predict. Sparse is used since our labels are integers (0-9) rather than one-hot encoded vectors. metrics=[\u0026lsquo;accuracy\u0026rsquo;]: We use accuracy to track the model’s performance during training and testing. Step 6: Train the Model # history = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=32, verbose=1) model.fit(): Train the model using the training data. validation_data=(X_test, y_test): Evaluate performance on the testing data during training. epochs=10: Train for 10 complete passes through the training dataset. batch_size=32: Update weights after every 32 samples. verbose=1: Print detailed information during training. Step 7: Evaluate the Model # test_loss, test_accuracy = model.evaluate(X_test, y_test) print(f\u0026#34;Test Accuracy: {test_accuracy:.2f}\u0026#34;) model.evaluate(X_test, y_test): Evaluates the model’s performance on the test data. test_accuracy: This gives us an idea of how well the model can classify unseen handwritten digits. Step 8: Visualize Training and Validation Performance (Optional) # # Convert history to DataFrame and plot accuracy and loss history_df = pd.DataFrame(history.history) history_df[[\u0026#39;accuracy\u0026#39;, \u0026#39;val_accuracy\u0026#39;]].plot() plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.title(\u0026#39;Training and Validation Accuracy\u0026#39;) plt.show() pd.DataFrame(history.history): Converts the training history to a DataFrame for easy visualization. Plotting: Training and validation accuracy are plotted to see if the model’s performance improves over epochs and whether it overfits or underfits. Step 9 (Optional): Make Some Predictions # # Make predictions on the first 5 test images predictions = model.predict(X_test[:5]) # Display the first 5 images with predicted and true labels for i in range(5): plt.imshow(X_test[i].reshape(28, 28), cmap=\u0026#39;gray\u0026#39;) plt.title(f\u0026#34;Predicted: {np.argmax(predictions[i])}, True: {y_test.iloc[i]}\u0026#34;) plt.axis(\u0026#39;off\u0026#39;) plt.show() model.predict(X_test[:5]): Predicts the labels for the first 5 test images. plt.imshow(): Displays each of the test images. np.argmax(predictions[i]): Retrieves the predicted label for each image. y_test.iloc[i]: Shows the true label. Video # Coming Soon. ","date":"2 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-2/","section":"Challenges","summary":"Today marks Day 2 of my 30 Days, 30 Deep Learning Projects Challenge. The task for today is to Classify handwritten digits using a simple NN on MNIST. Curious about how it went? Read on to see the results!","title":"Day 2: Classify handwritten digits using a simple NN on MNIST","type":"challenge"},{"content":"For this project, we\u0026rsquo;ll use a Feedforward Neural Network (NN) to predict the prices of houses based on various features such as number of rooms, area, location, etc. We\u0026rsquo;ll use Keras, a high-level API of TensorFlow, which makes building and training neural networks relatively easy.\nOutline of the Solution: # Dataset: We\u0026rsquo;ll use a sample dataset called California Housing Prices from the scikit-learn library to make things simple. This dataset has useful features for learning purposes. Steps: Load the dataset Preprocess the data Split into training and testing datasets Build a neural network model Train the model Evaluate the model Implementation # import pandas as pd from sklearn.datasets import fetch_california_housing from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense import matplotlib.pyplot as plt # Step 1: Load the data data = fetch_california_housing(as_frame=True) data_df = data.frame # Step 2: Build the feature and target datasets. X = data_df.drop(\u0026#39;MedHouseVal\u0026#39;, axis=1) # Feature Dataset y = data_df[\u0026#39;MedHouseVal\u0026#39;] # Target Dataset # Step 3: Split the dataset into training and validation datasets. # We will go with 80-20 ratio, training (80%) and validation (20%) X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) # Step 4: Scale the data scaler = StandardScaler() scaler.fit_transform(X_train) scaler.transform(X_val) # Step 5: Build the Neural Network Model model = Sequential() model.add( Dense(64, activation=\u0026#39;relu\u0026#39;, input_shape=(X.shape[1],)) ) model.add(Dense(32, activation=\u0026#39;relu\u0026#39;)) model.add(Dense(1)) # Step 6: Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;mse\u0026#39;, metrics=[\u0026#39;mae\u0026#39;]) # Step 7: Train the model history = model.fit(X_train, y_train, validation_split=0.2, epochs=50, batch_size=32, verbose=1) # Show the learning curve history_df = pd.DataFrame(history.history) history_df.loc[:, [\u0026#39;loss\u0026#39;, \u0026#39;val_loss\u0026#39;]].plot() plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.title(\u0026#39;Training and Validation loss across epochs\u0026#39;) plt.show() # Step 8: Evaluate the model test_loss = test_mae = model.evaluate(X_val, y_val) print(f\u0026#34;Validation Mean Absolute Error: {test_mae}\u0026#34;) # Output: # For Epochs - 50: # Validation Mean Absolute Error: [0.6920378804206848, 0.6737860441207886] # For Epochs- 10 # Validation Mean Absolute Error: [0.8164848685264587, 0.745993971824646] # For Epochs - 6 # Validation Mean Absolute Error: [94.31477355957031, 7.9173150062561035] # So 50 was a good number. Here’s the step-by-step explanation of each part of the code:\nStep 0: Importing Libraries # pandas as pd: Pandas is used to handle data in a structured format (DataFrames) which makes it easy to manipulate, explore, and clean data. fetch_california_housing: This function from sklearn allows us to load the California Housing dataset. train_test_split: This helps split the dataset into training and validation sets. StandardScaler: This is used to normalize the features to have zero mean and unit variance, which improves neural network performance. Sequential and Dense: These come from Keras (part of TensorFlow) and are used to define the structure of our neural network. matplotlib.pyplot as plt: This library is used to visualize the training history, such as the loss over epochs. Step 1: Load the Data # fetch_california_housing(as_frame=True): Loads the dataset into a Pandas DataFrame for better readability and data manipulation. data.frame: The dataset is stored as a DataFrame called data_df, allowing us to easily view and work with the features and target values. Step 2: Build the Feature and Target Datasets # X (Feature Dataset): Contains all the features that will be used to make predictions. We drop \u0026lsquo;MedHouseVal\u0026rsquo; as it’s the target variable (i.e., what we want to predict). y (Target Dataset): Contains the target variable, which is the Median House Value (MedHouseVal). This is what the model will try to predict. Step 3: Split the Dataset into Training and Validation Datasets # We split the data into training (80%) and validation (20%) sets. Training Data (X_train, y_train): Used to train the model. Validation Data (X_val, y_val): Used to evaluate the model’s performance on unseen data. random_state=42 ensures the split is reproducible. Step 4: Scale the Data # StandardScaler(): Initializes the scaler, which standardizes features by removing the mean and scaling to unit variance. scaler.fit_transform(X_train): Fits the scaler to X_train and transforms it. This step calculates the mean and variance of the training set and then scales the data accordingly. scaler.transform(X_val): Scales the validation set using the same mean and variance calculated from X_train. This ensures consistency between training and validation. Step 5: Build the Neural Network Model # Sequential(): Initializes a simple linear stack of layers. model.add(Dense(64, activation='relu', input_shape=(X.shape[1],))): Adds a Dense layer with 64 neurons and ReLU activation function. input_shape=(X.shape[1],) defines the shape of the input, which matches the number of features. model.add(Dense(32, activation='relu')): Adds another Dense layer with 32 neurons. model.add(Dense(1)): Adds the output layer with 1 neuron, which predicts the house price. Step 6: Compile the Model # optimizer='adam': Adam is a popular optimizer that adjusts the learning rate to optimize training speed. loss='mse': Mean Squared Error is used as the loss function, as this is a regression problem (predicting a continuous value). metrics=['mae']: We use Mean Absolute Error (MAE) as an additional metric to evaluate the performance of the model. Step 7: Train the Model # model.fit(): Trains the model using the training dataset. validation_split=0.2: During training, 20% of the training set will be used as validation data. epochs=50: The model will train for 50 complete passes through the entire training dataset. batch_size=32: The training dataset is divided into batches of 32 samples each, and the model will update its weights after each batch. verbose=1: Shows detailed logs during training. Show the Learning Curve # history.history: Contains the training and validation loss values collected during training. pd.DataFrame(history.history): Converts the history object into a Pandas DataFrame for easy visualization. .loc[:, ['loss', 'val_loss']].plot(): Plots the training loss and validation loss over the number of epochs. plt.xlabel('Epochs'), plt.ylabel('Loss'), plt.title(), plt.show(): Add labels, a title, and display the plot to visualize the model\u0026rsquo;s training and validation loss. Step 8: Evaluate the Model # model.evaluate(X_val, y_val): Evaluates the model on the validation set. test_loss: The Mean Squared Error on the validation set. test_mae: The Mean Absolute Error on the validation set. print(f\u0026quot;Validation Mean Absolute Error: {test_mae}\u0026quot;): Prints the validation Mean Absolute Error. Output and Observations: # Validation Mean Absolute Error for different epochs:\nWith 50 epochs, we got a reasonable MAE, which means the model performed well after sufficient training. With 10 epochs, the model\u0026rsquo;s MAE is higher, indicating it was under-trained and did not have enough epochs to learn the underlying patterns. With 6 epochs, the model\u0026rsquo;s error was quite high, indicating it had not trained enough to generalize. The observation is that training for 50 epochs worked better, as the model had enough time to learn the relationships in the data.\nVideo # ","date":"1 November 2024","externalUrl":null,"permalink":"/challenge/deep-learning/day-1/","section":"Challenges","summary":"Today marks Day 1 of my 30 Days, 30 Deep Learning Projects Challenge. The task for today is to Predict house prices using a feedforward neural network (NN). Curious about how it went? Read on to see the results!","title":"Day 1: Predict house prices using a feedforward neural network (NN)","type":"challenge"},{"content":"","date":"25 October 2024","externalUrl":null,"permalink":"/challenge/deep-learning/","section":"Challenges","summary":"List of Problems for 30 Days, 30 Deep Learning Projects Challenge","title":"30 Days, 30 Deep Learning Projects List","type":"challenge"},{"content":"After the amazing success of my 30 Days, 30 Machine Learning Projects Challenge, I’m excited to take on a new challenge!\nWhile completing the ML challenge, I found myself fascinated but occasionally overwhelmed by Deep Learning concepts. That’s when I knew I had to create a new challenge: 30 Days, 30 Deep Learning Projects.\nThis challenge is designed for gradual learning—starting with the basics of neural networks and ending with more advanced topics like GANs, Transformers, and BERT. I’ve carefully curated a list of projects, ensuring that the complexity builds up week by week. It’s time to dive deep!\nWeek 1: Neural Networks Fundamentals # Week Day Project Dataset Source 1 1 Predict house prices using a feedforward neural network (NN) Boston Housing Prices 1 2 Classify handwritten digits using a simple NN on MNIST MNIST Dataset 1 3 Explore backpropagation theory and tweak learning rates (MNIST) MNIST Dataset 1 4 Compare activation functions (ReLU, Sigmoid, Tanh) on Fashion MNIST Fashion MNIST 1 5 Experiment with optimizers (SGD, Adam, RMSprop) using a pre-built CNN on CIFAR-10 CIFAR-10 Dataset 1 6 Apply dropout and regularization (L2) for overfitting control (Titanic Dataset) Titanic Dataset 1 7 Fine-tune hyperparameters with Keras Tuner on a small NN Telco Customer Churn Dataset Week 2: CNNs and Computer Vision # Week Day Project Dataset Source 2 8 Build a simple CNN for CIFAR-10 image classification CIFAR-10 Dataset 2 9 Modify CNN with pooling layers and visualize filters CIFAR-100 Dataset 2 10 Use pre-built data augmentation methods in Keras on Fashion MNIST Fashion MNIST Dataset 2 11 Apply Transfer Learning with VGG16 for a simple classification task Cats vs Dogs Dataset 2 12 Implement YOLO for object detection (tutorial-based approach to simplify) Tutorial: YOLOv3 2 13 Explore image segmentation with U-Net for a small portion of Carvana dataset Carvana Image Masking Dataset 2 14 Mini-Project: Building a Custom CNN-based Student Model Using a Pre-Trained Teacher Model Use Kaggle Datasets Week 3: RNNs, LSTMs, and Time Series # Week Day Project Dataset Source 3 15 Prepare a simple time series dataset (Jena Climate or stock data) for RNN model Jena Climate Dataset 3 16 Build a basic RNN model for sequence prediction (temperature forecasting) Jena Climate Dataset 3 17 Build an LSTM model for sentiment analysis (IMDb Dataset) IMDb Reviews Dataset 3 18 Add attention mechanism to LSTM model for machine translation (split: theory on Day 18, code Day 19) English-French Dataset 3 19 Continue attention mechanism (implement and test it) English-French Dataset 3 20 Build an autoencoder-based anomaly detection system (part 1: data and model setup) Network Traffic Anomaly Dataset 3 21 Fine-tune and evaluate autoencoder model for anomaly detection Network Traffic Anomaly Dataset Week 4: GANs, Transformers, and Advanced Topics # Week Day Project Dataset Source 4 22 GAN Basics: Understand GAN architecture and set up the framework (on MNIST or Fashion MNIST) MNIST Dataset 4 23 Train and evaluate the GAN (continue from Day 22) Fashion MNIST Dataset 4 24 Build a Conditional GAN (CGAN) for generating specific images (Fashion MNIST) Fashion MNIST Dataset 4 25 Implement CycleGAN for style transfer (e.g., horse to zebra conversion) CycleGAN Dataset 4 26 Train the CycleGAN on a smaller image set (like horse2zebra) CycleGAN Dataset 4 27 Build a simple transformer-based model (BERT) for text classification (IMDb Dataset) IMDb Movie Reviews 4 28 Fine-tune the BERT model on a custom NLP task IMDb Movie Reviews 4 29 Work on SimCLR self-supervised learning or GPT-based text generation project (split into two parts) SimCLR Tutorial: SimCLR Paper, GPT-2: Hugging Face 4 30 Final Capstone: Finish any ongoing project or combine techniques for a final challenge (GANs, NLP) Explore Kaggle Competitions or Real-world Challenges Ready to Dive In? # I’m planning to start this Deep Learning challenge on 1st November, so I’ll take the time before then to brush up on the theory. I’ll also share additional resources and the reading list by 19th October(I have added them, checkout Resource list). If you’re ready to dive into deep learning, stay tuned, and feel free to join me on this journey!\nLet’s master Deep Learning one project at a time! 🚀\nPro Tip: Start small, stay consistent, and before you know it, you’ll be building complex models like a pro!\nResources List / Pre-requisite: # https://www.kaggle.com/learn/intro-to-deep-learning https://www.kaggle.com/learn/computer-vision https://www.kaggle.com/learn/machine-learning-explainability ","date":"11 October 2024","externalUrl":null,"permalink":"/post/30-days-30-deep-learning-projects-challenge/","section":"Post","summary":"After completing my 30 Days, 30 Machine Learning Projects Challenge, I’m ready for the next step: 30 Days, 30 Deep Learning Projects! This new challenge will guide me through deep learning, starting with the basics of neural networks and building up to advanced topics like GANs, Transformers, and BERT. I’ve designed the challenge to gradually increase in complexity, making it a perfect learning journey. If you’re ready to dive into deep learning, read moe to join me on this journey!","title":"30 Days, 30 Deep Learning Projects","type":"post"},{"content":"On Day 30, the final day of the 30-day machine learning challenge, I tackled the Capstone Project: Predicting loan approvals using ensemble learning with two powerful models—Random Forest and XGBoost. Ensemble learning combines the predictive power of multiple models to improve performance and accuracy.\nIf you want to see the code, you can find it here: GIT REPO.\nDataset: # I used the Loan Approval Prediction Dataset from Kaggle, which contains various features such as Applicant Income, Loan Amount, Loan Term, Education, Self-Employment Status, and Loan Status (approved or rejected). The goal was to predict whether a loan will be approved or rejected.\nSteps Taken: # Step 1: Load the Data\nI loaded the dataset and checked for any missing values and inconsistencies. The dataset didn’t have any missing values, but I noticed that some columns and values had extra spaces, which were removed for consistency. Step 2: Preprocessing the Data\nI cleaned up the column names and values by stripping the leading and trailing spaces from the features. I used One-Hot Encoding to convert categorical features such as Education and Self-Employed into numerical values. The target column, Loan Status, was converted into 0 for \u0026ldquo;Rejected\u0026rdquo; and 1 for \u0026ldquo;Approved\u0026rdquo; for consistency. Step 3: Splitting the Data\nI split the dataset into training and validation sets (80% training, 20% validation). Step 4: Model Training\nRandom Forest: I trained the Random Forest model using 100 decision trees. Each tree makes a prediction based on a random subset of data, and the final prediction is made based on the majority vote. XGBoost (Extreme Gradient Boosting): I trained the XGBoost model with a learning rate of 1.1 and 100 boosting iterations. XGBoost sequentially builds trees, each correcting the errors made by the previous ones. Step 5: Model Evaluation\nBoth models performed exceptionally well, with 98% accuracy on the validation set. Here\u0026rsquo;s a breakdown of their performance:\nRandom Forest:\nAccuracy: 97.89% Confusion Matrix: [[309, 9], [9, 527]] Precision/Recall: Both models achieved nearly equal precision and recall, with a balanced performance across both approved and rejected loans. XGBoost:\nAccuracy: 97.89% Confusion Matrix: [[310, 10], [8, 526]] Precision/Recall: XGBoost showed slightly better recall on rejected loans, making it just as effective as Random Forest. Step 6: Visualization\nI plotted the confusion matrices for both models using heatmaps, which showed how well each model predicted approved and rejected loans. Code Implementation: # import pandas as pd from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestClassifier from xgboost import XGBClassifier from sklearn.metrics import accuracy_score, confusion_matrix, classification_report import matplotlib.pyplot as plt import seaborn as sns # Step 1: Load the data. data = pd.read_csv(\u0026#39;dataset/loan_approval_dataset.csv\u0026#39;) # print(data.info()) # print(data.head()) # Step 2: Preprocess the data # Check for missing values #print(data.isnull().sum()) # We do not have any missing values. # Find the categorical data. # print(data[\u0026#39; education\u0026#39;].unique()) # [\u0026#39; Graduate\u0026#39; \u0026#39; Not Graduate\u0026#39;] # print(data[\u0026#39; self_employed\u0026#39;].unique()) # [\u0026#39; No\u0026#39; \u0026#39; Yes\u0026#39;] # print(data[\u0026#39; loan_status\u0026#39;].unique()) # [\u0026#39; Approved\u0026#39; \u0026#39; Rejected\u0026#39;] # The dataset has an extra space before the actual value. # Update the column name first. data.columns = data.columns.str.strip() # Now the values. data[\u0026#39;education\u0026#39;] = data[\u0026#39;education\u0026#39;].str.strip() data[\u0026#39;self_employed\u0026#39;] = data[\u0026#39;self_employed\u0026#39;].str.strip() data[\u0026#39;loan_status\u0026#39;] = data[\u0026#39;loan_status\u0026#39;].str.strip() # Convert the categorical data into numeric using One-Hot Encoding data = pd.get_dummies(data, columns=[\u0026#39;education\u0026#39;, \u0026#39;self_employed\u0026#39;], drop_first=False) # For consistency, model expects in 0, 1. Change Rejected as 0 and Approved as 1 data[\u0026#39;loan_status\u0026#39;] = data[\u0026#39;loan_status\u0026#39;].replace({\u0026#39;Approved\u0026#39;: 1, \u0026#39;Rejected\u0026#39;: 0}) # [1 0] # Create features and Target Dataset X = data.drop(\u0026#39;loan_status\u0026#39;, axis=1) # Feature Dataset y = data[\u0026#39;loan_status\u0026#39;] # Target Dataset # Step 3: Split the datasets into training and validation datasets. X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) # Step 4: Build and Train the Models # Random Forest Model # How it works: It creates multiple decision trees on different subsets of the data, and each tree gives a prediction. # The final prediction is based on the majority vote (classification) or the average (regression) of all the trees. model_rf = RandomForestClassifier(n_estimators=100, random_state=42) model_rf.fit(X_train, y_train) # XGBoost (Extreme Gradient Boosting): # How it works: XGBoost adds trees one by one, with each tree attempting to minimize the errors made by the previous # trees using a gradient descent approach. It uses boosting techniques to improve performance. model_xgb = XGBClassifier(n_estimators=100, learning_rate=1.1, random_state=42) model_xgb.fit(X_train, y_train) # Step 5: Make Predictions and Evaluate # Random Forest predictions_rf = model_rf.predict(X_val) accuracy_score_rf = accuracy_score(predictions_rf, y_val) confusion_matrix_rf = confusion_matrix(predictions_rf, y_val) classification_report_rf = classification_report(predictions_rf, y_val) print(\u0026#34;Random Forest: \u0026#34;) print(f\u0026#34;Accuracy Score: {accuracy_score_rf}\u0026#34;) print(f\u0026#34;Confusion Matrix: {confusion_matrix_rf}\u0026#34;) print(f\u0026#34;Classification Report: {classification_report_rf}\u0026#34;) # XGBoost predictions_xgb = model_xgb.predict(X_val) accuracy_score_xbg = accuracy_score(predictions_xgb, y_val) confusion_matrix_xgb = confusion_matrix(predictions_xgb, y_val) classification_report_xgb = classification_report(predictions_xgb, y_val) print(\u0026#34;XGBoost: \u0026#34;) print(f\u0026#34;Accuracy Score: {accuracy_score_xbg}\u0026#34;) print(f\u0026#34;Confusion Matrix: {confusion_matrix_xgb}\u0026#34;) print(f\u0026#34;Classification Report: {classification_report_xgb}\u0026#34;) # Step 6: Visualization plt.figure(figsize=(7, 5)) sns.heatmap(confusion_matrix_rf, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=[\u0026#39;Approved\u0026#39;, \u0026#39;Rejected\u0026#39;], yticklabels=[\u0026#39;Approved\u0026#39;, \u0026#39;Rejected\u0026#39;]) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Random Forest Confusion Matrix\u0026#39;) plt.figure(figsize=(7, 5)) sns.heatmap(confusion_matrix_xgb, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=[\u0026#39;Approved\u0026#39;, \u0026#39;Rejected\u0026#39;], yticklabels=[\u0026#39;Approved\u0026#39;, \u0026#39;Rejected\u0026#39;]) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;XGBoost Confusion Matrix\u0026#39;) plt.show() HeatMap # Both Random Forest and XGBoost delivered excellent results, each achieving 98% accuracy. The confusion matrix heatmaps made it easy to see how well each model predicted both approved and rejected loans, with only minor differences between the two models.\nGratitude: # This project wraps up the 30-day Machine Learning challenge! 🎉 It’s been such a great experience—it really helped me stay consistent, and seeing all the compiled code made me feel super productive. Setting a measurable goal like this was awesome, and I’m so pumped about the outcomes that I’m planning to take on even more!\nWhile tackling the problems, I realized I had a tough time with the Deep Learning challenges. So, I’ve decided to dive into another adventure: the \u0026ldquo;30 Days, 30 Deep Learning Projects Challenge\u0026rdquo;!\nI’m still working on the problem list and figuring out how it’ll all come together, but I’ll have all the details ready soon.\nStay tuned for the next chapter!\n","date":"10 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-30/","section":"Challenges","summary":"On the final day, I built Random Forest and XGBoost models to predict loan approvals. The Random Forest model achieved an 84% accuracy, with a good balance between predicting approvals and rejections. XGBoost reached 82.14% accuracy, excelling in approvals but less accurate for rejections. Both models were visualized through confusion matrix heatmaps, showcasing the strength of ensemble learning in predictive modeling.","title":"Day 30 - Capstone Project: Predicting Loan Approvals Using Ensemble Learning (Random Forest, XGBoost)","type":"challenge"},{"content":"On Day 29, I focused on building models to predict credit risk using both Logistic Regression and Support Vector Machines (SVM). The dataset contains various financial and personal details of individuals applying for credit, and the goal was to predict whether an applicant poses a credit risk.\nIf you want to see the code, you can find it here: GIT REPO.\nDataset: # I used Germen Credit Risk dataset from Kaggle, which includes various features like Age, Sex, Job, Housing, Saving accounts, Checking accounts, Credit amount, Duration, and Purpose. The task was to predict whether a loan applicant is at credit risk (target: Risk = 1) or not (target: Risk = 0).\nCode # # Problem: Credit risk prediction with Logistic Regression and SVM # Dataset: https://www.kaggle.com/datasets/uciml/german-credit import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.svm import SVC from sklearn.metrics import accuracy_score, confusion_matrix, classification_report # Step 1: Load the data data = pd.read_csv(\u0026#39;dataset/german_credit_data.csv\u0026#39;) # Step 2: Preprocess the data # Remove the first column as it is unnamed in the csv file. data = data.iloc[:, 1:] # print(data.isnull().sum()) # Check the count of missing values in dataset # Saving accounts has 183 missing values. # Checking account has 394 missing values. data[\u0026#39;Saving accounts\u0026#39;].fillna(\u0026#39;unknown\u0026#39;, inplace=True) data[\u0026#39;Checking account\u0026#39;].fillna(\u0026#39;unknown\u0026#39;, inplace=True) # Convert the categorical columns into numeric using One-Hot Encoding data = pd.get_dummies(data, columns=[\u0026#39;Sex\u0026#39;, \u0026#39;Housing\u0026#39;, \u0026#39;Saving accounts\u0026#39;, \u0026#39;Checking account\u0026#39;, \u0026#39;Purpose\u0026#39;], drop_first=True) # Step 3: Create Feature and Target datasets # Define a simple rule of generating risk column # If the account has credit amount of 5000 and the Duration is more than 24 hours, it is considered a high risk. data[\u0026#39;risk\u0026#39;] = ((data[\u0026#39;Credit amount\u0026#39;] \u0026gt; 5000) \u0026amp; (data[\u0026#39;Duration\u0026#39;] \u0026gt; 24)).astype(int) X = data.drop(\u0026#39;risk\u0026#39;, axis=1) # Features y = data[\u0026#39;risk\u0026#39;] # Target # Step 4: Split the data into training and validation datasets X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) # Step 5: Feature Scaling scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_val_scaled = scaler.transform(X_val) # Step 6: Build and Train the Logistic Regression Model model_log_reg = LogisticRegression() model_log_reg.fit(X_train_scaled, y_train) # Step 7: Build and Train the SVM Model model_svm = SVC() model_svm.fit(X_train_scaled, y_train) # Step 8: Make Prediction and Evaluate Logistic Regression Model log_reg_predictions = model_log_reg.predict(X_val_scaled) accuracy_score_lg = accuracy_score(log_reg_predictions, y_val) confusion_matrix_lg = confusion_matrix(log_reg_predictions, y_val) classification_report_log_reg = classification_report(log_reg_predictions, y_val) print(\u0026#34;Logistic Regression: \u0026#34;) print(f\u0026#34;Accuracy Score: {accuracy_score_lg}\u0026#34;) print(f\u0026#34;Confusion Matrix: \\n {confusion_matrix_lg}\u0026#34;) print(f\u0026#34;Classification Report: \\n {classification_report_log_reg}\u0026#34;) # Step 9: Make Prediction and Evaluate the SVM Model svm_predictions = model_svm.predict(X_val_scaled) accuracy_score_svm = accuracy_score(svm_predictions, y_val) confusion_matrix_svm = confusion_matrix(svm_predictions, y_val) classification_report_svm = classification_report(svm_predictions, y_val) print(\u0026#34;SVM Model: \u0026#34;) print(f\u0026#34;Accuracy Score: {accuracy_score_svm}\u0026#34;) print(f\u0026#34;Confusion Matrix: \\n {confusion_matrix_svm}\u0026#34;) print(f\u0026#34;Classification Report: \\n {classification_report_svm}\u0026#34;) Understand the code: # Step 1: Load the Data\nI loaded the dataset and removed the first unnamed index column, as it wasn\u0026rsquo;t needed for modeling. Step 2: Data Preprocessing\nHandling Missing Values: The Saving accounts column had 183 missing values, and the Checking accounts column had 394 missing values. I filled these missing values with \u0026lsquo;unknown\u0026rsquo;. One-Hot Encoding: I converted categorical columns such as Sex, Housing, Saving accounts, Checking account, and Purpose into numerical values using One-Hot Encoding. Step 3: Create Feature and Target Datasets\nI created a simple risk rule where: If a Credit amount is greater than 5000 and the Duration is more than 24 months, the applicant is considered a high risk. The target variable is labeled as risk, and all other columns are used as features. Step 4: Split the Data\nI split the dataset into training and validation sets, with 80% of the data used for training and 20% for validation. Step 5: Feature Scaling\nI applied StandardScaler to normalize the features, ensuring both the Logistic Regression and SVM models would perform optimally. Step 6: Build and Train the Models\nI trained both the Logistic Regression and SVM models using the scaled training data. Step 7 \u0026amp; 8: Evaluation of Logistic Regression\nLogistic Regression: Accuracy Score: 0.965 Confusion Matrix: [[175 6] [ 1 18]] Classification Report: precision recall f1-score support 0 0.99 0.97 0.98 181 1 0.75 0.95 0.84 19 accuracy 0.96 200 macro avg 0.87 0.96 0.91 200 weighted avg 0.97 0.96 0.97 200 SVM Model: Accuracy Score: 0.96 Confusion Matrix: [[176 8] [ 0 16]] Classification Report: precision recall f1-score support 0 1.00 0.96 0.98 184 1 0.67 1.00 0.80 16 accuracy 0.96 200 macro avg 0.83 0.98 0.89 200 weighted avg 0.97 0.96 0.96 200 Key Insights: # Logistic Regression achieved high precision (0.99) and recall (0.97) for predicting non-risky loans (class 0). It also performed well in detecting risky loans (class 1) with a recall of 0.95. SVM showed perfect recall (1.00) for identifying all risky loans (class 1), but had a slightly lower precision (0.67), meaning it was more likely to misclassify non-risky cases as risky. Both models showed high accuracy (96%+), but Logistic Regression had a better balance between precision and recall. Gratitude # What a blast revisiting the models on a new dataset! It’s Day 29 of the challenge, and guess what? Tomorrow is the BIG day! 🎉\nStay Tuned!\n","date":"9 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-29/","section":"Challenges","summary":"On Day 29, I built and compared two models—Logistic Regression and SVM—for credit risk prediction. The Logistic Regression model achieved a high accuracy of 96.5%, with a well-balanced precision and recall for predicting both risky and non-risky loans. The SVM model also performed well, achieving 96% accuracy but with a slightly lower precision for risky loans. Logistic Regression showed a better balance between precision and recall, while SVM exhibited perfect recall for risky loans. Both models demonstrated strong performance, making them suitable for credit risk prediction tasks.","title":"Day 29 - Credit Risk Prediction with Logistic Regression and SVM","type":"challenge"},{"content":"On Day 28 of the challenge, I tackled the task of building a simple chatbot using traditional NLP techniques. The goal was to implement the chatbot in two ways:\nWithout Vectorization – Using a simple rule-based approach with pattern matching. With Vectorization – Using Bag of Words (BoW) and Cosine Similarity to improve the chatbot\u0026rsquo;s flexibility. If you want to see the code, you can find it here: GIT REPO.\nApproach 1: Without Vectorization # In this approach, I implemented the chatbot using basic pattern matching without vectorization. The chatbot attempts to match user inputs with predefined patterns and respond based on those matches. This method uses Bag of Words (BoW) directly by comparing the words in the user’s input with the patterns.\nSteps Taken: # Define Intents: Created a dictionary of intents where each intent had patterns (common ways the user might phrase a query) and responses (how the chatbot should reply). Preprocess Input: The user input is converted to lowercase, punctuation is removed, and it\u0026rsquo;s tokenized into words to prepare it for pattern matching. Pattern Matching: For each user input, the chatbot compares the all the words with the predefined patterns for each intent and returns the most relevant intent. Generate Responses: Once an intent is matched, the chatbot picks a random response from the predefined responses for that intent. Create Chat loop: Now, we need to create a while loop where the chatbot keeps asking for input and responds until the user says something like \u0026ldquo;bye\u0026rdquo; or \u0026ldquo;exit\u0026rdquo;. Code:\n# Problem: Build a simple chatbot using traditional NLP techniques # Without vectorization import re import random # Step 1: Create a dataset of intents and responses intents = { \u0026#39;greeting\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;hello\u0026#39;, \u0026#39;hi\u0026#39;, \u0026#39;hey\u0026#39;, \u0026#39;good morning\u0026#39;, \u0026#39;good evening\u0026#39;, \u0026#39;good afternoon\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Hello!\u0026#39;, \u0026#39;Hi there!\u0026#39;, \u0026#39;Greetings!\u0026#39;, \u0026#39;Good day!\u0026#39;, \u0026#39;Hey! How can I help you today?\u0026#39;] }, \u0026#39;farewell\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;bye\u0026#39;, \u0026#39;goodbye\u0026#39;, \u0026#39;see you later\u0026#39;, \u0026#39;farewell\u0026#39;, \u0026#39;take care\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Goodbye!\u0026#39;, \u0026#39;Take care!\u0026#39;, \u0026#39;See you later!\u0026#39;, \u0026#39;Farewell!\u0026#39;, \u0026#39;Have a great day!\u0026#39;] }, \u0026#39;thanks\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;thank you\u0026#39;, \u0026#39;thanks\u0026#39;, \u0026#39;thank you so much\u0026#39;, \u0026#39;much appreciated\u0026#39;, \u0026#39;thanks a lot\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;You’re welcome!\u0026#39;, \u0026#39;Glad I could help!\u0026#39;, \u0026#39;Anytime!\u0026#39;, \u0026#39;My pleasure!\u0026#39;, \u0026#39;No problem!\u0026#39;] }, \u0026#39;bot_name\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;what is your name\u0026#39;, \u0026#39;who are you\u0026#39;, \u0026#39;tell me your name\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I’m your friendly chatbot!\u0026#39;, \u0026#39;You can call me Chatbot!\u0026#39;, \u0026#39;I am a chatbot created to help you.\u0026#39;] }, \u0026#39;bot_purpose\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;what can you do\u0026#39;, \u0026#39;how can you help me\u0026#39;, \u0026#39;what do you do\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I can assist you with basic queries, answer questions, and chat with you!\u0026#39;, \u0026#39;I’m here to help you with anything you need.\u0026#39;, \u0026#39;I can chat with you and answer simple questions!\u0026#39;] }, \u0026#39;feeling\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;how are you\u0026#39;, \u0026#39;how are you doing\u0026#39;, \u0026#39;are you okay\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I’m just a bot, but I’m doing great!\u0026#39;, \u0026#39;I’m feeling helpful today!\u0026#39;, \u0026#39;I’m here to help you, so I’m doing well!\u0026#39;] }, \u0026#39;age\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;how old are you\u0026#39;, \u0026#39;what is your age\u0026#39;, \u0026#39;when were you created\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I don’t have an age like humans, but I’m always learning!\u0026#39;, \u0026#39;Age is just a number, and I don’t have one!\u0026#39;, \u0026#39;I was created recently to help you out!\u0026#39;] }, \u0026#39;weather\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;what is the weather\u0026#39;, \u0026#39;how is the weather today\u0026#39;, \u0026#39;tell me the weather\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I can’t check the weather right now, but you can check your weather app!\u0026#39;, \u0026#39;I don’t have access to weather data, but I hope it’s nice outside!\u0026#39;, \u0026#39;Check your weather app for accurate information!\u0026#39;] }, \u0026#39;joke\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;tell me a joke\u0026#39;, \u0026#39;make me laugh\u0026#39;, \u0026#39;tell a joke\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Why did the computer go to the doctor? Because it had a virus!\u0026#39;, \u0026#39;Why don’t robots have brothers? Because they all have trans-sisters!\u0026#39;, \u0026#39;I’d tell you a joke about UDP, but you might not get it!\u0026#39;] }, \u0026#39;help\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;help me\u0026#39;, \u0026#39;i need help\u0026#39;, \u0026#39;can you help me\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Sure, I’m here to help! What do you need?\u0026#39;, \u0026#39;Of course! Let me know how I can assist you.\u0026#39;, \u0026#39;I’m happy to help. Please tell me what you need assistance with!\u0026#39;] }, \u0026#39;unknown\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;who is the president\u0026#39;, \u0026#39;where is the moon\u0026#39;, \u0026#39;how to cook pasta\u0026#39;, \u0026#39;tell me a story\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Sorry, I’m not sure how to answer that.\u0026#39;, \u0026#39;I don’t have the answer to that right now.\u0026#39;, \u0026#39;Hmm, I don’t know, but I can find out!\u0026#39;] } } # Step 2: Preprocess user input def preprocess(text): text = text.lower() # Conver to lowercase text = re.sub(r\u0026#39;[^\\w\\s]\u0026#39;, \u0026#39;\u0026#39;, text) # Remove punctuation tokens = text.split() # Tokenization by splitting return tokens # Step 3: Implement pattern matching with bag of words (BOW) def match_intent(processed_input, intents): for intent, intent_data in intents.items(): patterns = intent_data[\u0026#39;patterns\u0026#39;] for pattern in patterns: processed_pattern = preprocess(pattern) if all(word in processed_input for word in processed_pattern): return intent return None # Step 4: Generate the response def get_response(intent, intents): return random.choice(intents[intent][\u0026#39;responses\u0026#39;]) def chatbot(): print(\u0026#34;Hey! I am a 28_chatbot, How can i help you? Type bye to exit\u0026#34;) while True: user_input = input(\u0026#34;You: \u0026#34;) if user_input.lower() == \u0026#39;bye\u0026#39;: print(\u0026#34;Chatbot: Good Bye!\u0026#34;) break processed_input = preprocess(user_input) matched_intent = match_intent(processed_input=processed_input, intents=intents) if matched_intent: response = get_response(intent=matched_intent, intents=intents) print(f\u0026#34;Chatbot: {response}\u0026#34;) else: print(f\u0026#34;Chatbot: Sorry i could not understand that.\u0026#34;) # Run the chatbot chatbot() Example Interaction:\nYou: Hey Chatbot: Good day! You: How can you help me Chatbot: I can chat with you and answer simple questions! You: Who is the president Chatbot: Sorry, I’m not sure how to answer that. You: Who will win Chatbot: Sorry i could not understand that. You: bye Chatbot: Good Bye! Approach 2: With Vectorization # In this second approach, I enhanced the chatbot using CountVectorizer from scikit-learn to vectorize the input and the patterns. This allowed for more flexible matching using cosine similarity between user input and predefined patterns.\nSteps Taken: # Vectorization of Patterns: I used CountVectorizer to transform the predefined patterns into vectors (Bag of Words). Vectorize User Input: When the user inputs a sentence, it is vectorized using the same CountVectorizer that was trained on the patterns. Cosine Similarity: I used Cosine Similarity to compare the user input vector with all pattern vectors, identifying the most similar pattern and its corresponding intent. Generate Responses: After matching the most similar intent, the chatbot responds with a random response from that intent’s list of responses. Code:\n# Problem: Build a simple chatbot using traditional NLP techniques # With Vectorization import random from sklearn.feature_extraction.text import CountVectorizer from sklearn.metrics.pairwise import cosine_similarity # Step 1: Create a dataset of intents and responses intents = { \u0026#39;greeting\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;hello\u0026#39;, \u0026#39;hi\u0026#39;, \u0026#39;hey\u0026#39;, \u0026#39;good morning\u0026#39;, \u0026#39;good evening\u0026#39;, \u0026#39;good afternoon\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Hello!\u0026#39;, \u0026#39;Hi there!\u0026#39;, \u0026#39;Greetings!\u0026#39;, \u0026#39;Good day!\u0026#39;, \u0026#39;Hey! How can I help you today?\u0026#39;] }, \u0026#39;farewell\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;bye\u0026#39;, \u0026#39;goodbye\u0026#39;, \u0026#39;see you later\u0026#39;, \u0026#39;farewell\u0026#39;, \u0026#39;take care\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Goodbye!\u0026#39;, \u0026#39;Take care!\u0026#39;, \u0026#39;See you later!\u0026#39;, \u0026#39;Farewell!\u0026#39;, \u0026#39;Have a great day!\u0026#39;] }, \u0026#39;thanks\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;thank you\u0026#39;, \u0026#39;thanks\u0026#39;, \u0026#39;thank you so much\u0026#39;, \u0026#39;much appreciated\u0026#39;, \u0026#39;thanks a lot\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;You’re welcome!\u0026#39;, \u0026#39;Glad I could help!\u0026#39;, \u0026#39;Anytime!\u0026#39;, \u0026#39;My pleasure!\u0026#39;, \u0026#39;No problem!\u0026#39;] }, \u0026#39;bot_name\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;what is your name\u0026#39;, \u0026#39;who are you\u0026#39;, \u0026#39;tell me your name\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I’m your friendly chatbot!\u0026#39;, \u0026#39;You can call me Chatbot!\u0026#39;, \u0026#39;I am a chatbot created to help you.\u0026#39;] }, \u0026#39;bot_purpose\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;what can you do\u0026#39;, \u0026#39;how can you help me\u0026#39;, \u0026#39;what do you do\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I can assist you with basic queries, answer questions, and chat with you!\u0026#39;, \u0026#39;I’m here to help you with anything you need.\u0026#39;, \u0026#39;I can chat with you and answer simple questions!\u0026#39;] }, \u0026#39;feeling\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;how are you\u0026#39;, \u0026#39;how are you doing\u0026#39;, \u0026#39;are you okay\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I’m just a bot, but I’m doing great!\u0026#39;, \u0026#39;I’m feeling helpful today!\u0026#39;, \u0026#39;I’m here to help you, so I’m doing well!\u0026#39;] }, \u0026#39;age\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;how old are you\u0026#39;, \u0026#39;what is your age\u0026#39;, \u0026#39;when were you created\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I don’t have an age like humans, but I’m always learning!\u0026#39;, \u0026#39;Age is just a number, and I don’t have one!\u0026#39;, \u0026#39;I was created recently to help you out!\u0026#39;] }, \u0026#39;weather\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;what is the weather\u0026#39;, \u0026#39;how is the weather today\u0026#39;, \u0026#39;tell me the weather\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;I can’t check the weather right now, but you can check your weather app!\u0026#39;, \u0026#39;I don’t have access to weather data, but I hope it’s nice outside!\u0026#39;, \u0026#39;Check your weather app for accurate information!\u0026#39;] }, \u0026#39;joke\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;tell me a joke\u0026#39;, \u0026#39;make me laugh\u0026#39;, \u0026#39;tell a joke\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Why did the computer go to the doctor? Because it had a virus!\u0026#39;, \u0026#39;Why don’t robots have brothers? Because they all have trans-sisters!\u0026#39;, \u0026#39;I’d tell you a joke about UDP, but you might not get it!\u0026#39;] }, \u0026#39;help\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;help me\u0026#39;, \u0026#39;i need help\u0026#39;, \u0026#39;can you help me\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Sure, I’m here to help! What do you need?\u0026#39;, \u0026#39;Of course! Let me know how I can assist you.\u0026#39;, \u0026#39;I’m happy to help. Please tell me what you need assistance with!\u0026#39;] }, \u0026#39;unknown\u0026#39;: { \u0026#39;patterns\u0026#39;: [\u0026#39;who is the president\u0026#39;, \u0026#39;where is the moon\u0026#39;, \u0026#39;how to cook pasta\u0026#39;, \u0026#39;tell me a story\u0026#39;], \u0026#39;responses\u0026#39;: [\u0026#39;Sorry, I’m not sure how to answer that.\u0026#39;, \u0026#39;I don’t have the answer to that right now.\u0026#39;, \u0026#39;Hmm, I don’t know, but I can find out!\u0026#39;] } } vectorizer = CountVectorizer() # Train the vectorizer on all patterns all_patterns = [] intent_labels = [] for intent, intent_data in intents.items(): patterns = intent_data[\u0026#39;patterns\u0026#39;] all_patterns.extend(patterns) # Combine all patterns intent_labels.extend([intent] * len(patterns)) # Track of which pattern belongs to which intent # Fit the vectorizer X = vectorizer.fit_transform(all_patterns) # Build the vocubulary on all the patterns. # Function to find the best matching intent def match_intent(user_input): user_vec = vectorizer.transform([user_input]) similarity_scores = cosine_similarity(user_vec, X) # Compare user input to all patterns best_matching_intent_idx = similarity_scores.argmax() # Get the maximum value simarity index return intent_labels[best_matching_intent_idx] def chatbot(): print(\u0026#34;Hey, How can i help you? Type bye to exit\u0026#34;) while True: user_input = input(\u0026#34;You: \u0026#34;) if user_input.lower() == \u0026#39;bye\u0026#39;: print(\u0026#34;Good bye!\u0026#34;) break matched_intent = match_intent(user_input) responce = random.choice(intents[matched_intent][\u0026#39;responses\u0026#39;]) print(f\u0026#34;Chatbot: {responce}\u0026#34;) # Run the chatbot chatbot() Example Interaction:\nHey, How can i help you? Type bye to exit You: Hey Chatbot: Hi there! You: How can you help me Chatbot: I’m here to help you with anything you need. You: Who is the president Chatbot: Sorry, I’m not sure how to answer that. You: Who will win Chatbot: I am a chatbot created to help you. You: bye Good bye! Improvements with Vectorization: # Flexible Matching: The use of cosine similarity allowed the chatbot to match user input more flexibly, even if the words used weren\u0026rsquo;t exactly the same as the predefined patterns. Scalability: This approach scales better for larger datasets, as it can handle more complex inputs and varied phrasing Gratitude # On Day 28, I implemented two versions of a chatbot using traditional NLP techniques:\nWithout Vectorization – A basic rule-based approach with exact word matching. With Vectorization – A more flexible approach using CountVectorizer and cosine similarity for better input matching. Learning the basics is always fun, especially to understand how even the simplest forms of technology, like ChatGPT, are built.\nStay Tuned!\n","date":"8 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-28/","section":"Challenges","summary":"This day’s focus on building a chatbot highlighted the importance of foundational NLP techniques. By comparing a simple rule-based method with a more sophisticated vectorized approach, I gained valuable insights into how even the most basic chatbot functionalities are developed, laying the groundwork for more advanced projects in the future.","title":"Day 28 - Building a Simple Chatbot Using Traditional NLP Techniques","type":"challenge"},{"content":"The task for Day 27 was to build a Convolutional Neural Network (CNN) for image classification on the CIFAR-10 dataset. The CIFAR-10 dataset consists of 60,000 32x32 color images in 10 different classes, such as airplanes, cars, birds, and cats. The goal was to create a small CNN to classify these images into the correct categories and evaluate its performance.\nIf you want to see the code, you can find it here: GIT REPO.\nDataset: # The CIFAR-10 dataset contains:\n50,000 training images and 10,000 test images. Images are 32x32 pixels with 3 color channels (RGB). Each image belongs to one of 10 classes. Steps Taken: # Step 1: Load and Preprocess the Data # I loaded the CIFAR-10 dataset using TensorFlow’s built-in dataset utility. The images were normalized by scaling the pixel values from 0-255 to a range of [0, 1] to ensure that the model learns efficiently. The labels were one-hot encoded, which converted the categorical labels into binary vectors for use in multi-class classification.\n(X_train, y_train), (X_val, y_val) = cifar10.load_data() X_train = X_train.astype(\u0026#39;float32\u0026#39;) / 255.0 X_val = X_val.astype(\u0026#39;float32\u0026#39;) / 255.0 y_train = to_categorical(y_train, 10) y_val = to_categorical(y_val, 10) Step 2: Build a Small CNN Architecture # I created a Sequential model with 3 convolutional layers followed by MaxPooling layers to reduce spatial dimensions. Each convolutional layer used ReLU activation, and the number of filters increased with each layer to capture more complex features.\nI used a Dense (fully connected) layer with 128 units and Dropout to prevent overfitting. Finally, the model used a softmax output layer to output probabilities for each of the 10 classes.\nmodel = Sequential() model.add(Conv2D(32, (3, 3), activation=\u0026#39;relu\u0026#39;, input_shape=(32, 32, 3))) model.add(MaxPooling2D(2, 2)) model.add(Conv2D(64, (3, 3), activation=\u0026#39;relu\u0026#39;)) model.add(MaxPooling2D(2, 2)) model.add(Conv2D(128, (3, 3), activation=\u0026#39;relu\u0026#39;)) model.add(MaxPooling2D(2, 2)) model.add(Flatten()) model.add(Dense(128, activation=\u0026#39;relu\u0026#39;)) model.add(Dropout(0.5)) model.add(Dense(10, activation=\u0026#39;softmax\u0026#39;)) Step 3: Compile the Model # I compiled the model using Adam optimizer and categorical crossentropy as the loss function (because it’s a multi-class classification problem). The model was also configured to track accuracy as a performance metric.\nmodel.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;]) Step 4: Train the Model # The model was trained for 10 epochs with a batch size of 64 using 80% of the data for training and 20% for validation. During training, both the accuracy and loss were tracked for both the training and validation sets.\nhistory = model.fit(X_train, y_train, epochs=10, batch_size=64, validation_data=(X_val, y_val)) Step 5: Evaluate the Model and Visualize the Results # After training, I evaluated the model on the test set. The validation accuracy reached 72.58%, indicating that the model performed reasonably well for a basic CNN architecture on CIFAR-10.\nModel Architecture and Performance:\nEpoch 1/10: val_accuracy: 0.5161 Epoch 5/10: val_accuracy: 0.6925 Epoch 10/10: val_accuracy: 0.7258 Validation Accuracy: 0.7258 The training and validation curves show consistent improvement, with no significant overfitting observed. Here are the accuracy and loss plots over the epochs:\nval_loss, val_acc = model.evaluate(X_val, y_val) print(f\u0026#34;Validation Accuracy: {val_acc}\u0026#34;) plt.figure(figsize=(10, 6)) # Plot accuracy plt.subplot(1, 2, 1) plt.plot(history.history[\u0026#39;accuracy\u0026#39;], label=\u0026#34;Training Accuracy\u0026#34;) plt.plot(history.history[\u0026#39;val_accuracy\u0026#39;], label=\u0026#34;Validation Accuracy\u0026#34;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Accuracy\u0026#39;) plt.legend() # Plot loss plt.subplot(1, 2, 2) plt.plot(history.history[\u0026#39;loss\u0026#39;], label=\u0026#34;Training Loss\u0026#34;) plt.plot(history.history[\u0026#39;val_loss\u0026#39;], label=\u0026#34;Validation Loss\u0026#34;) plt.xlabel(\u0026#39;Epochs\u0026#39;) plt.ylabel(\u0026#39;Loss\u0026#39;) plt.legend() plt.show() Step 6: Make Predictions # I randomly selected an image from the test set and had the model make a prediction. In this case, the model predicted class 5 (dog), and the actual class was also class 5, which was correct.\n# Random image from the test set random_idx = np.random.randint(0, len(X_val)) random_image = X_val[random_idx] # Make prediction prediction = model.predict(np.expand_dims(random_image, axis=0)) prediction_class = np.argmax(prediction, axis=1) # Print the actual and predicted class print(f\u0026#34;Actual Class: {np.argmax(y_val[random_idx])}\u0026#34;) print(f\u0026#34;Predicted Class: {prediction_class[0]}\u0026#34;) # Display the image with the predicted class plt.imshow(random_image) plt.title(f\u0026#34;Predicted: {prediction_class[0]}\u0026#34;) plt.show() Outout:\nPrediction array: [[6.4272822e-06 1.4340281e-07 5.5310270e-03 6.6405848e-02 4.2907866e-03 8.9277107e-01 1.0174847e-03 2.9964956e-02 3.4941979e-07 1.2036602e-05]] Actual Class of the random image: 5 Prediction class: 5 The model predicted correctly for this example, but there were cases where it made mistakes (e.g., predicting class 2 instead of 4).\nModel Performance: # Validation Accuracy: ~71% The model performed reasonably well, with training and validation accuracy steadily improving over the 10 epochs. The loss values for both the training and validation sets decreased, and there was no sign of overfitting. Gratitude # This was the first problem on CNN and the second in deep learning. I\u0026rsquo;m finding it a bit complex to digest in a single day. As I mentioned yesterday, I plan to take on another challenge focused on deep learning after completing this one. And let’s be honest, I can’t wait to wrap up this challenge and reward myself with a delicious dinner—because coding deserves a tasty celebration!\u0026quot;\nStay Tuned!\n","date":"7 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-27/","section":"Challenges","summary":"On Day 27, I created a CNN model for classifying CIFAR-10 images. After training for 10 epochs, the model reached 72.58% accuracy on the validation set. The model is performing decently, but there’s potential for improvement with more training or enhancements like data augmentation.","title":"Day 27 - Image Classification with a Small CNN on CIFAR-10 Dataset","type":"challenge"},{"content":"Today\u0026rsquo;s task was to use LSTM (Long Short-Term Memory), a type of Recurrent Neural Network (RNN), to forecast electricity consumption based on historical data. This project served as an introduction to deep learning for time series forecasting, where LSTMs excel at capturing long-term dependencies in sequential data.\nIf you want to see the code, you can find it here: GIT REPO.\nDataset: # I used the AEP Hourly Dataset from Kaggle\u0026rsquo;s Hourly Energy Consumption Dataset problem, which consists of 121,273 hourly electricity consumption records in megawatts (MW). The goal was to predict future electricity consumption based on past hourly data.\nCode Flow: # Load and inspect the dataset. Convert the Datetime column to the correct format. Handle missing values (if any). Normalize the AEP_MW column for better model performance. Create sequences for the LSTM model (sliding windows). Split the data into training and test sets. Train the LSTM model and make predictions. Visualize the results. Step 1: Load and Convert Datetime # We\u0026rsquo;ll start by loading the dataset and converting the Datetime column to a datetime object so that we can handle it easily in the future if we need to. This will allow us to index by time if needed.\nimport pandas as pd # Load the data data = pd.read_csv(\u0026#39;dataset/AEP_hourly.csv\u0026#39;) # Convert \u0026#39;Datetime\u0026#39; column to datetime object data[\u0026#39;Datetime\u0026#39;] = pd.to_datetime(data[\u0026#39;Datetime\u0026#39;]) # Set the \u0026#39;Datetime\u0026#39; column as the index data.set_index(\u0026#39;Datetime\u0026#39;, inplace=True) # Check for any missing values print(data.isnull().sum()) Step 2: Handle Missing Values (if any) # Check if there are any missing values. If there are, we can either drop them or use interpolation to fill them.\n# Handle missing values by interpolating data = data.interpolate() # Drop any remaining missing values data = data.dropna() If there are no missing values, you can skip this step.\nStep 3: Normalize the Data # It\u0026rsquo;s crucial to normalize the AEP_MW values to a range between 0 and 1. This helps the LSTM model converge faster and perform better.\nfrom sklearn.preprocessing import MinMaxScaler # Select the \u0026#39;AEP_MW\u0026#39; column for normalization scaler = MinMaxScaler(feature_range=(0, 1)) scaled_data = scaler.fit_transform(data[[\u0026#39;AEP_MW\u0026#39;]]) # Check the first few rows of the scaled data print(scaled_data[:5]) Step 4: Create Sequences for the LSTM Model # We’ll use sliding windows of, for example, 60 hours to predict the next hour of electricity consumption. The create_sequences function will help us transform the data into the format needed for LSTM.\nimport numpy as np # data: The entire time series data (in this case, the scaled electricity consumption data). # time_steps: The number of previous time steps (hours) you want to use as input to predict the next time step. def create_sequences(data, time_steps): sequences = [] # This list will store the input sequences (i.e., the previous 60 hours of electricity consumption). target = [] # This list will store the corresponding target values. for i in range(len(data) - time_steps): sequences.append(data[i: i + time_steps]) target.append(data[i + time_steps]) return np.array(sequences), np.array(target) # Set time_steps to 60 (using the previous 60 hours to predict the next hour) time_steps = 60 # Create sequences X, y = create_sequences(scaled_data, time_steps) # Check the shape of the sequences print(X.shape, y.shape) Step 5: Split the Data into Training and Test Sets # We’ll split the data into 80% training and 20% test sets to ensure the model is trained on historical data and tested on future data.\n# Split the data into training and test sets (80% train, 20% test) train_size = int(len(X) * 0.8) X_train, X_test = X[:train_size], X[train_size:] y_train, y_test = y[:train_size], y[train_size:] # Check the shape of the train/test sets print(X_train.shape, X_test.shape) Step 6: Build and Train the LSTM Model # Now, we can build a simple LSTM model using Keras. We’ll use 50 units in the LSTM layer, followed by a dense layer to output the predicted value.\nimport tensorflow as tf from tensorflow.keras.models import Sequential from tensorflow.keras.layers import LSTM, Dense # Create the LSTM model model = Sequential() # Add an LSTM layer with 50 units model.add(LSTM(50, return_sequences=False, input_shape=(X_train.shape[1], 1))) # Add a Dense layer to output a single value (the predicted consumption) model.add(Dense(1)) # Compile the model model.compile(optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;mean_squared_error\u0026#39;) # Train the model history = model.fit(X_train, y_train, epochs=20, batch_size=32, validation_data=(X_test, y_test)) Step 7: Evaluate the Model # Once the model is trained, we can evaluate it by making predictions on the test set.\n# Make prediction predictions = model.predict(X_val) # Transforms the predicted values from the scaled range (0-1) back to the original range, which represents real # electricity consumption values in MW. predictions = scaler.inverse_transform(predictions) y_val_org = scaler.inverse_transform(y_val.reshape(-1, 1)) # Visualization plt.figure(figsize=(10, 7)) plt.plot(y_val_org, label=\u0026#34;Actual Consumption\u0026#34;) plt.plot(predictions, label=\u0026#34;Predicted Consumption\u0026#34;) plt.xlabel(\u0026#39;Time\u0026#39;) plt.ylabel(\u0026#39;Consumption (MW)\u0026#39;) plt.title(\u0026#39;Electricity Consumption Forecasting\u0026#39;) plt.legend() plt.show() The plot showed the actual and predicted electricity consumption values overlapping closely, indicating that the LSTM model captured the underlying patterns quite well. Both short-term and long-term trends were effectively modeled, with only minor deviations.\nResults: # The model was able to closely track the actual electricity consumption, with minor prediction errors, as seen in the plot where the orange line (predicted consumption) closely followed the blue line (actual consumption). The overall performance suggests that the LSTM model is well-suited for this type of time series forecasting problem.\nGratitude # Being the first problem in deep learning, it was exciting to understand and explore new topics. I plan to start a dedicated challenge on deep learning after this one finishes. I also can’t wait to tackle all the problems in this challenge.\nStay Tuned!\n","date":"6 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-26/","section":"Challenges","summary":"On Day 26, I used LSTM (Long Short-Term Memory) to build a model for forecasting hourly electricity consumption. Using past 60 hours of consumption data as input, the model was able to predict the next hour’s consumption with high accuracy. After training, the model’s predictions closely followed the actual electricity demand in the test set, demonstrating the effectiveness of LSTM for time series forecasting.","title":"Day 26- Time Series Forecasting of Electricity Consumption using LSTM (Intro to Deep Learning)","type":"challenge"},{"content":"For today\u0026rsquo;s task, I performed Sentiment Analysis on customer reviews using traditional Natural Language Processing (NLP) techniques. The goal was to classify customer reviews from the IMDb Movie Reviews Dataset as either positive or negative using preprocessing, feature extraction with TF-IDF, and a simple Logistic Regression model.\nIf you want to see the code, you can find it here: GIT REPO.\nDataset: # I used the IMDb Dataset of 50K Movie Reviews, which contains 50,000 reviews, each labeled as either positive or negative.\nSteps Taken: # Let\u0026rsquo;s import the libraries first.\nimport pandas as pd import re from nltk.tokenize import word_tokenize from nltk.corpus import stopwords from nltk.stem import PorterStemmer from sklearn.model_selection import train_test_split from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score, confusion_matrix, classification_report import matplotlib.pyplot as plt import seaborn as sns Step 1: Load the Data # I started by loading the dataset and inspecting the first few rows. The dataset contains two columns: review (text data) and sentiment (positive/negative labels).\ndata = pd.read_csv(\u0026#39;dataset/IMDB_Dataset.csv\u0026#39;) Step 2: Data Preprocessing # Next, I preprocessed the text data to clean and normalize it for analysis. The preprocessing steps included:\nRemoving HTML tags and special characters. Lowercasing the text. Tokenizing the text into words. Removing stopwords (common words that don’t contribute much to meaning, e.g., \u0026ldquo;the\u0026rdquo;, \u0026ldquo;and\u0026rdquo;). Stemming the words using PorterStemmer, which reduces words to their root form (e.g., \u0026ldquo;running\u0026rdquo; to \u0026ldquo;run\u0026rdquo;). This was done to reduce noise in the text and focus on the core words contributing to sentiment.\ndef preprocesss_text(text): # Remove the HTMl tags and special characters. text = re.sub(r\u0026#39;\u0026lt;.*?\u0026gt;\u0026#39;, \u0026#39;\u0026#39;, text) # \u0026lt;html\u0026gt; \u0026lt;body\u0026gt; etc text = re.sub(r\u0026#39;[^\\w\\s]\u0026#39;, \u0026#39;\u0026#39;, text) # Convert to lower. text = text.lower() # Tokenize tokens = word_tokenize(text) # Remove the stopwords and apply stemming. stop_words = set(stopwords.words(\u0026#39;english\u0026#39;)) tokens = [word for word in tokens if word not in stop_words] # Remove stopwords stemmer = PorterStemmer() tokens = [stemmer.stem(word) for word in tokens] # Applied stemming. return \u0026#39; \u0026#39;.join(tokens) # Apply preprocessing on the reviews feature data[\u0026#39;cleaned_review\u0026#39;] = data[\u0026#39;review\u0026#39;].apply(preprocesss_text) Step 3: Create Features and Target Datasets # After preprocessing the text, I created two main components:\nFeatures (X): The cleaned reviews (text after preprocessing). Target (y): The sentiment (positive/negative) corresponding to each review. These will be used to train and evaluate the model.\nX = data[\u0026#39;cleaned_review\u0026#39;] # Feature: Cleaned reviews y = data[\u0026#39;sentiment\u0026#39;] # Target: Sentiment (positive/negative) Step 4: Split the Data into Training and Validation Sets # I split the data into training (80%) and validation (20%) sets to evaluate the model\u0026rsquo;s performance. This ensures that the model is trained on one part of the data and tested on unseen data to measure generalization.\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) Step 5: Feature Extraction Using TF-IDF # To convert the text into numerical form, I used TF-IDF (Term Frequency-Inverse Document Frequency), which captures the importance of words in the corpus. I applied the TF-IDF vectorizer on the training data and transformed both the training and validation sets.\ntfidf_vectorizer = TfidfVectorizer(max_features=5000) X_train_tfidf = tfidf_vectorizer.fit_transform(X_train) X_val_tfidf = tfidf_vectorizer.transform(X_val) Step 6: Train the Model # I used Logistic Regression to build the sentiment analysis model. Logistic Regression is a popular choice for text classification tasks due to its simplicity and effectiveness.\nmodel = LogisticRegression() model.fit(X_train_tfidf, y_train) Step 7: Make Prediction Evaluate the Model # After training the model, I made predictions on the validation set and evaluated its performance using accuracy, a confusion matrix, and a classification report (precision, recall, F1-score).\npredictions = model.predict(X_val_tfidf) accuracy_score = accuracy_score(y_val, predictions) confusion_matrix = confusion_matrix(y_val, predictions) classification_report = classification_report(y_val, predictions) print(f\u0026#34;Accuracy Score: \\n {accuracy_score}\u0026#34;) print(f\u0026#34;Confusion Matrix: \\n {confusion_matrix}\u0026#34;) print(f\u0026#34;Classification Report: \\n {classification_report}\u0026#34;) Output\n# Output # Accuracy Score: # 0.8848 # Confusion Matrix: # [[4304 657] # [ 495 4544]] # Classification Report: # precision recall f1-score support # # negative 0.90 0.87 0.88 4961 # positive 0.87 0.90 0.89 5039 # # accuracy 0.88 10000 # macro avg 0.89 0.88 0.88 10000 # weighted avg 0.89 0.88 0.88 10000 Accuracy: The model achieved an accuracy of 88.48% on the validation set.\nConfusion Matrix: Showed that the model correctly classified most reviews, with some false positives and false negatives.\nThe model correctly classified 4304 negative reviews and 4544 positive reviews. There were 657 false positives (where the model incorrectly classified negative reviews as positive) and 495 false negatives (where the model incorrectly classified positive reviews as negative). Classification Report: Provided detailed metrics like precision, recall, and F1-score for both positive and negative sentiments.\nNegative Precision = 0.90: Out of all reviews predicted as negative, 90% were actually negative. Positive Precision = 0.87: Out of all reviews predicted as positive, 87% were actually positive. Negative Recall = 0.87: Out of all actual negative reviews, 87% were correctly predicted as negative. Positive Recall = 0.90: Out of all actual positive reviews, 90% were correctly predicted as positive. Negative F1-Score = 0.88: The F1-score for negative reviews shows a balance between precision (0.90) and recall (0.87). Positive F1-Score = 0.89: The F1-score for positive reviews also shows a good balance between precision (0.87) and recall (0.90). Negative Support = 4961: There were 4961 actual negative reviews in the validation set. Positive Support = 5039: There were 5039 actual positive reviews in the validation set. Step 8: Visualization # I visualized the confusion matrix to get a clearer understanding of how the model performed. The confusion matrix shows how many reviews were correctly classified as positive or negative, and where the model made errors.\nplt.figure(figsize=(7, 5)) sns.heatmap(confusion_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=[\u0026#39;Negative\u0026#39;,\u0026#39;Positive\u0026#39;], yticklabels=[\u0026#39;Negative\u0026#39;,\u0026#39;Positive\u0026#39;]) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Confusion Matrix\u0026#39;) plt.show() Gratitude # Although we have performed all the above steps in many of our previous problems, this being the first problem in NLP, it was great to learn what is needed to get started. I\u0026rsquo;m looking forward to the next problem.\nStay Tuned!\n","date":"5 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-25/","section":"Challenges","summary":"This project marked our introduction to NLP, where we revisited familiar steps while gaining new insights into the specific requirements for NLP tasks. The experience was enriching, and I’m excited to apply this knowledge to upcoming challenges.","title":"Day 25 - Sentiment Analysis of Customer Reviews Using Traditional NLP Techniques","type":"challenge"},{"content":"Today\u0026rsquo;s challenge was to use K-Means clustering to segment customers based on their behavioral patterns such as age, annual income, and spending score. This technique is widely used in customer segmentation to create distinct groups of customers with similar behaviors, helping businesses tailor marketing strategies and improve customer experience.\nIf you want to see the code, you can find it here: GIT REPO.\nDataset: # For this project, I used the Mall Customer Segmentation Dataset from Kaggle. The dataset contains the following columns:\nCustomerID: Unique identifier for each customer. Gender: Gender of the customer (Male/Female). Age: Customer\u0026rsquo;s age. Annual Income (k$): The customer\u0026rsquo;s annual income in thousands. Spending Score (1-100): A score assigned based on the customer\u0026rsquo;s spending behavior. Steps for Implementation: # Data Preprocessing:\nSince K-Means relies on numerical features, we will drop the CustomerID and Gender columns (or encode the Gender column if you\u0026rsquo;d like to include it). Scale the features, as K-Means clustering is sensitive to the scale of data. Applying K-Means Clustering:\nPerform K-Means clustering using the features Age, Annual Income, and Spending Score. Visualize the clusters. Use the Elbow Method to find the optimal number of clusters. Interpret the Clusters:\nUnderstand what each cluster represents (e.g., high spenders vs. low spenders). Here’s how you can implement it:\nimport pandas as pd from sklearn.preprocessing import StandardScaler from sklearn.cluster import KMeans import matplotlib.pyplot as plt import seaborn as sns # Step 1: Load the data data = pd.read_csv(\u0026#39;dataset/Mall_Customers.csv\u0026#39;) # print(data.info()) # print(data.head()) # Output # # Column Non-Null Count Dtype # --- ------ -------------- ----- # 0 CustomerID 200 non-null int64 # 1 Gender 200 non-null object # 2 Age 200 non-null int64 # 3 Annual Income (k$) 200 non-null int64 # 4 Spending Score (1-100) 200 non-null int64 # dtypes: int64(4), object(1) # memory usage: 7.9+ KB # None # CustomerID Gender Age Annual Income (k$) Spending Score (1-100) # 0 1 Male 19 15 39 # 1 2 Male 21 15 81 # 2 3 Female 20 16 6 # 3 4 Female 23 16 77 # 4 5 Female 31 17 40 # Step 2: Data Preprocessing # Drop CustomerID as it is not useful for clustering. data.drop(\u0026#39;CustomerID\u0026#39;, axis=1) # Encode Gender, male - 0, female - 1 data[\u0026#39;Gender\u0026#39;] = data[\u0026#39;Gender\u0026#39;].map({\u0026#39;Male\u0026#39;: 0, \u0026#39;Female\u0026#39;: 1}) # Handle missing value. #print(data.isnull().sum()) # We don\u0026#39;t have any missing value. # Step 3: Create features for Cluster X = data[[\u0026#39;Gender\u0026#39;, \u0026#39;Age\u0026#39;, \u0026#39;Annual Income (k$)\u0026#39;, \u0026#39;Spending Score (1-100)\u0026#39;]] # Step 4: Apply Scaling scaler = StandardScaler() X_scaled = scaler.fit_transform(X) # Step 5: Apply Elbow method to find the optimal number of clusters (K) inertia = [] for k in range(1, 11): kmeans = KMeans(n_clusters=k, init=\u0026#39;k-means++\u0026#39;, random_state=42, n_init=\u0026#39;auto\u0026#39;) kmeans.fit(X_scaled) inertia.append(kmeans.inertia_) # What k-means++ Does: The k-means++ initialization method is a smart way of choosing the initial centroids. # Instead of choosing random points, k-means++ spreads out the initial centroids as follows: # # It first randomly selects one centroid from the data points. # Then, for each remaining centroid, it selects the next centroid from the data points that are far from the # already chosen centroids. The probability of choosing a data point as a centroid is proportional to its # distance from the nearest already chosen centroid. # Plot the Elbow curve plt.figure(figsize=(7, 5)) plt.plot(range(1, 11), inertia, marker=\u0026#39;o\u0026#39;, linestyle=\u0026#39;--\u0026#39;) plt.title(\u0026#39;Elbow Curve\u0026#39;) plt.xlabel(\u0026#39;K value\u0026#39;) plt.ylabel(\u0026#39;Inertia\u0026#39;) plt.show() # Step 6: Apply K Means on the optimal number of clusters (K = 4) optimal_k = 4 kmeans = KMeans(n_clusters=optimal_k, init=\u0026#39;k-means++\u0026#39;, random_state=42, n_init=\u0026#39;auto\u0026#39;) y_kmeans = kmeans.fit_predict(X_scaled) # Step 7: Visualize the clusters plt.figure(figsize=(10, 7)) sns.scatterplot(x=X_scaled[:, 2], y=X_scaled[:, 3], hue=y_kmeans, palette=\u0026#34;viridis\u0026#34;, s=100) # We are plotting the Annual Income (x-axis) against the Spending Score (y-axis) for each customer. plt.title(\u0026#39;Customer Segments (K Means Cluster)\u0026#39;) plt.xlabel(\u0026#39;Annual Income\u0026#39;) plt.ylabel(\u0026#39;Spending Score\u0026#39;) plt.show() # Step 8: Add the clusters information for each row to original data data[\u0026#39;Clusters\u0026#39;] = y_kmeans print(data) # Output # CustomerID Gender Age Annual Income (k$) Spending Score (1-100) Clusters # 0 1 0 19 15 39 2 # 1 2 0 21 15 81 2 # 2 3 1 20 16 6 3 # 3 4 1 23 16 77 3 # 4 5 1 31 17 40 3 # .. ... ... ... ... ... ... # 195 196 1 35 120 79 3 # 196 197 1 45 126 28 1 # 197 198 0 32 126 74 2 # 198 199 0 32 137 18 1 # 199 200 0 30 137 83 2 Explanation: # Feature Scaling: We scale the features because K-Means uses Euclidean distance, and differences in the scale of features can impact clustering results. Elbow Method: We use the Elbow Method to determine the optimal number of clusters by plotting the within-cluster sum of squares (WCSS) and looking for the \u0026ldquo;elbow\u0026rdquo; point, where adding more clusters doesn’t significantly improve the fit. K-Means Clustering: We apply K-Means clustering with the number of clusters determined from the elbow plot (in this case, it’s set to 5). Visualization: A scatter plot is used to visualize the clusters formed by K-Means, where Annual Income and Spending Score are the features plotted against each other, and different colors represent different customer segments. Cluster Assignment: Finally, the Cluster labels are added to the original dataset, allowing you to analyze and interpret the customer segments. See how the graph looks:\nGratitude # This is the second problem involving KMeans. I felt confident with the topics and I\u0026rsquo;m really excited about my progress. Looking forward to the next day!\nStay Tuned!\n","date":"4 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-24/","section":"Challenges","summary":"We built a customer segmentation model using K-Means clustering, selecting 4 clusters based on the Elbow Method. After preprocessing and scaling the features, the model successfully segmented customers based on their annual income and spending score, revealing distinct customer groups with similar behaviors. This segmentation can be used by businesses to personalize marketing strategies and enhance customer experience.","title":"Day 24 - K-Means Clustering to Segment Customers Based on Behavior","type":"challenge"},{"content":"Today’s challenge involved detecting fraudulent transactions using two machine learning algorithms: Logistic Regression and Random Forest. Fraud detection is critical in the financial world, where minimizing losses from fraudulent transactions is a top priority. This problem posed a unique challenge due to the highly imbalanced dataset, where only a small percentage of transactions are actually fraudulent.\nIf you want to see the code, you can find it here: GIT REPO.\nDataset: # I used the Credit Card Fraud Detection Dataset from Kaggle, which contains 284,807 transactions, of which only 492 are labeled as fraudulent. The dataset includes numerical features generated through PCA transformations (to protect sensitive information), as well as the feature \u0026lsquo;Class\u0026rsquo;, which represents whether the transaction is fraudulent (1) or legitimate (0).\nSteps Taken: # Let\u0026rsquo;s import the required libraries first.\nimport pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score, confusion_matrix, classification_report from sklearn.ensemble import RandomForestClassifier import matplotlib.pyplot as plt import seaborn as sns Step 1: Load and Explore the Dataset # I began by loading the dataset and inspecting its structure. The dataset is highly imbalanced, with fraud making up only 0.17% of all transactions. Here\u0026rsquo;s how I loaded and explored the data:\n# Load the dataset data = pd.read_csv(\u0026#39;dataset/creditcard.csv\u0026#39;) print(data.info()) print(data.head()) Step 2: Data Preprocessing # The dataset didn’t contain any missing values, so I could proceed directly to separating the features and the target variable. Given the importance of feature scaling for Logistic Regression, I used StandardScaler to normalize the features.\n# Handle missing values. print(data.isnull().sum()) # We dont have any column with null value. # Separate features and target X = data.drop(\u0026#39;Class\u0026#39;, axis=1) # Features (everything except \u0026#39;Class\u0026#39;) y = data[\u0026#39;Class\u0026#39;] # Target (fraud label: 1 for fraud, 0 for legitimate) # Split into training and testing datasets X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) # Feature scaling scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_val_scaled = scaler.transform(X_val) Step 3: Logistic Regression # I trained a Logistic Regression model on the scaled data. Logistic Regression is a simple and interpretable model that works well for binary classification tasks.\n# Train Logistic Regression model log_reg_model = LogisticRegression() log_reg_model.fit(X_train_scaled, y_train) # Predict on validation data log_reg_predictions = log_reg_model.predict(X_val_scaled) # Evaluate the model\u0026#39;s performance log_reg_accuracy = accuracy_score(y_val, log_reg_predictions) log_reg_conf_matrix = confusion_matrix(y_val, log_reg_predictions) log_reg_classfication_report = classification_report(y_val, log_reg_predictions) print(f\u0026#34;Accuracy Score of Logistic Regression: {log_reg_accuracy}\u0026#34;) print(f\u0026#34;Confusion Matrix of Logistic Regression: {log_reg_conf_matrix}\u0026#34;) print(f\u0026#34;Classification Report of Logistic Regression: {log_reg_classfication_report}\u0026#34;) Logistic Regression Results:\nAccuracy: The model achieved an accuracy score of 0.999, which indicates that most transactions were classified correctly. Precision and Recall: While the precision for fraud detection was strong, the recall was somewhat lower, meaning the model struggled to identify all fraudulent transactions. Confusion Matrix: The confusion matrix highlighted that out of 98 fraudulent transactions, 41 were missed. Step 4: Random Forest # Next, I trained a Random Forest classifier, which is a more powerful model, well-suited to handling the complexity of this dataset and the class imbalance.\n# Train Random Forest model ran_for_model = RandomForestClassifier(random_state=42) ran_for_model.fit(X_train, y_train) # Predict on validation data ran_for_predictions = ran_for_model.predict(X_val) # Evaluate the model\u0026#39;s performance ran_for_accuracy = accuracy_score(y_val, ran_for_predictions) ran_for_conf_matrix = confusion_matrix(y_val, ran_for_predictions) ran_for_classfication_report = classification_report(y_val, ran_for_predictions) print(f\u0026#34;Accuracy Score of Random Forest: {ran_for_accuracy}\u0026#34;) print(f\u0026#34;Confusion Matrix of Random Forest: {ran_for_conf_matrix}\u0026#34;) print(f\u0026#34;Classification Report of Random Forest: {ran_for_classfication_report}\u0026#34;) Random Forest Results:\nAccuracy: The Random Forest model achieved a slightly higher accuracy score of 0.9995. Precision and Recall: The model showed excellent precision and recall, significantly improving its ability to detect fraudulent transactions. The recall was much higher than in Logistic Regression, meaning it identified most of the fraud cases. Confusion Matrix: The Random Forest model missed fewer fraud cases compared to Logistic Regression, with only 23 false negatives out of 98 fraud cases. Step 5: Visualizing Results # Finally, I visualized the confusion matrices for both models using Seaborn, which helped me better understand how the models were performing in classifying fraudulent and legitimate transactions.\n# Confusion Matrix for Logistic Regression plt.figure(figsize=(7, 5)) sns.heatmap(log_reg_conf_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=[\u0026#39;Legitimate\u0026#39;, \u0026#39;Fraud\u0026#39;], yticklabels=[\u0026#39;Legitimate\u0026#39;, \u0026#39;Fraud\u0026#39;]) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Confusion Matrix for Logistic Regression\u0026#39;) plt.show() # Confusion Matrix for Random Forest plt.figure(figsize=(7, 5)) sns.heatmap(ran_for_conf_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=[\u0026#39;Legitimate\u0026#39;, \u0026#39;Fraud\u0026#39;], yticklabels=[\u0026#39;Legitimate\u0026#39;, \u0026#39;Fraud\u0026#39;]) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Confusion Matrix for Random Forest\u0026#39;) plt.show() Results: # Logistic Regression:\nAccuracy: 99.91% Precision: 0.86 for fraud detection Recall: 0.58 (The model missed 41 fraud cases out of 98) Key Takeaway: While Logistic Regression performed well overall, it struggled to catch all fraud cases, especially in such an imbalanced dataset.\nRandom Forest:\nAccuracy: 99.96% Precision: 0.97 for fraud detection Recall: 0.77 (The model missed only 23 fraud cases out of 98) Key Takeaway: Random Forest performed significantly better in identifying fraudulent transactions, especially in terms of recall, which is critical in fraud detection.\nConfusion Matrix # The confusion matrices provided visual insight into how the models classified transactions. Random Forest was much better at reducing false negatives (missed fraud cases) compared to Logistic Regression.\nGratitude # We saw that:\nLogistic Regression: A simple, interpretable model, but it struggled with imbalanced data, missing a number of fraud cases. Random Forest: A more powerful model that excelled at detecting fraud, even in a highly imbalanced dataset. Its higher recall meant it was better suited to fraud detection where identifying all fraud cases is crucial. Stay Tuned!\n","date":"3 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-23/","section":"Challenges","summary":"We built two models—Logistic Regression and Random Forest—to detect fraudulent financial transactions. While Logistic Regression achieved high accuracy, it struggled with recall, missing many fraud cases. Random Forest, on the other hand, performed significantly better, with improved precision and recall, making it more effective at identifying fraud in this highly imbalanced dataset.","title":"Day 23 - Fraud Detection in Financial Transactions Using Logistic Regression and Random Forest","type":"challenge"},{"content":"Today, I built a Recommender System using Matrix Factorization to predict how users will rate items they haven’t interacted with. I used Singular Value Decomposition (SVD), a popular matrix factorization technique, to decompose the user-item interaction matrix and predict missing ratings. The goal was to recommend movies to users by predicting the ratings they might give to movies they haven’t watched yet.\nIf you want to see the code, you can find it here: GIT REPO.\nUnderstanding Dataset: # I used the MovieLens small dataset, which contains user ratings for various movies. This dataset is widely used in recommendation system projects because it contains user-item interactions, which are perfect for collaborative filtering.\nStep-by-Step Approach: # Step 1: Loading the Dataset # First, I loaded the ratings.csv file, which contains user ratings for different movies. It has the following columns:\nuserId: Unique identifier for each user. movieId: Unique identifier for each movie. rating: The rating given by the user to the movie (scale from 0.5 to 5.0). # Step 1: Load the MovieLens dataset. ratings = pd.read_csv(\u0026#39;dataset/ratings.csv\u0026#39;) print(ratings.head()) Step 2: Preparing Data for the Surprise Library # To use the Surprise library, which is specifically designed for recommendation systems, I needed to format the data properly:\nReader: Defines the scale of ratings (0.5 to 5.0 in this case). Dataset.load_from_df(): Converts the pandas DataFrame into the format required by Surprise for further processing. # Step 2: Prepare the data for Surprise library. reader = Reader(rating_scale=(ratings[\u0026#39;rating\u0026#39;].min(), ratings[\u0026#39;rating\u0026#39;].max())) data = Dataset.load_from_df(ratings[[\u0026#39;userId\u0026#39;, \u0026#39;movieId\u0026#39;, \u0026#39;rating\u0026#39;]], reader) Step 3: Splitting the Data for Training and Validation # I split the dataset into training and validation sets to evaluate the model\u0026rsquo;s performance. The training set is used to build the model, and the validation set is used to test how well the model generalizes to unseen data.\n# Step 3: Split data into train-validation datasets trainset, valset = train_test_split(data, test_size=0.2) Step 4: Applying SVD for Matrix Factorization # SVD (Singular Value Decomposition) is a matrix factorization technique that helps break down a large user-item matrix into lower-dimensional matrices. This technique uncovers latent factors that represent hidden relationships between users and items.\nSVD decomposes the matrix into three smaller matrices: one for users, one for items, and one diagonal matrix of singular values. The model can then make predictions by reconstructing the matrix and predicting the missing values (i.e., ratings). To evaluate the model, I used cross-validation, which splits the dataset into different parts, trains the model on some parts, and tests it on others. I measured the model\u0026rsquo;s accuracy using RMSE (Root Mean Squared Error) and MSE (Mean Squared Error).\n# Step 4: Matrix Factorization using (Singular Value Decomposition) SVD svd = SVD() cross_validate(svd, data, measures=[\u0026#39;RMSE\u0026#39;, \u0026#39;MSE\u0026#39;], cv=5, verbose=True) Cross-validation helps evaluate the performance of the model across multiple splits of the data, ensuring that the model generalizes well.\nCross-validation results:\nRMSE measures the average magnitude of prediction error. Lower values indicate better predictions. MSE measures the average squared difference between the predicted and actual ratings. Fold 1 RMSE: 0.8724 MSE: 0.7611 Fold 2 RMSE: 0.8690 MSE: 0.7552 Fold 3 RMSE: 0.8734 MSE: 0.7628 Fold 4 RMSE: 0.8805 MSE: 0.7752 Fold 5 RMSE: 0.8733 MSE: 0.7627 Step 5: Training the Model on the Full Dataset # Once I evaluated the model using cross-validation, I retrained the SVD model on the entire dataset to maximize the amount of data the model sees.\n# Step 5: Train the model. trainset_full = data.build_full_trainset() svd.fit(trainset_full) Step 6: Making Predictions # After training, I used the model to predict how user 1 would rate movie 6, a movie they had previously rated. The prediction was very close to the actual rating:\n# Step 6: Make Predictions user_id = 1 movie_id = 6 prediction = svd.predict(user_id, movie_id) print(f\u0026#34;Prediction for user {user_id} and movie {movie_id} is: {prediction}\u0026#34;) Output:\nPrediction for user 1 and movie 6 is: user: 1 item: 6 r_ui = None est = 4.49 {\u0026#39;was_impossible\u0026#39;: False} The model predicted a rating of 4.49, which is close to the actual rating of 4 given by user 1 for movie 6.\nStep 7: Evaluating the Model # Finally, I evaluated the model on the validation set by calculating the RMSE. The RMSE for the validation set was 0.641, indicating that the model\u0026rsquo;s predictions are quite close to the actual ratings.\n# Step 7: Evaluate the model on validation data val_predictions = svd.test(valset) rmse = accuracy.rmse(val_predictions) print(f\u0026#34;Root mean square error for validation data: {rmse}\u0026#34;) Output:\nRoot mean square error for validation data: 0.6410624356100669 Results # The RMSE of 0.641 on the validation set indicates that the model\u0026rsquo;s predictions are off by around 0.641 rating points on average, which is a good level of accuracy given that the ratings are on a scale from 0.5 to 5.0.\nGratitude # It\u0026rsquo;s the first day of exploring advanced models. I couldn\u0026rsquo;t grasp the details of how SVD works and why it is effective. I plan to cover this topic in depth after the challenge.\nStay tuned for the next problem!\n","date":"2 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-22/","section":"Challenges","summary":"It marks Day 22 of my 30 Days, 30 Machine Learning Projects Challenge. Today’s challenge was build Recommender System With Matrix Factorization. Curious about how it went? Read on to see the results!","title":"Day 22 - Recommender System with Matrix Factorization","type":"challenge"},{"content":"Today’s challenge was focused on deploying a machine learning model for real-time predictions. I chose to deploy our Day 8 problem of Fake News Detection model using FastAPI as the API framework and Heroku for hosting the app.\nIf you want to see the code, you can find it here: GIT REPO.\nTools \u0026amp; Technologies # FastAPI: For building the API. Heroku: For deploying the app. Scikit-learn: For the machine learning model (PassiveAggressive Classifier and TfidfVectorizer). Pickle: To save and load the trained model and vectorizer. Uvicorn: ASGI server for running FastAPI apps. The Model # For this project, I used a PassiveAggressive Classifier with a TfidfVectorizer to detect fake news articles. The dataset included two categories: True News and Fake News. Here\u0026rsquo;s a step-by-step guide on how to complete this task, starting with training the model (we already explored this in the Day 8 problem, so please refer to that post before proceeding), followed by creating a FastAPI app and deploying it on Heroku.\nSteps for Full Flow: # Step 1: Finalize and Save the Model # Since you have already built and trained the model, the next step is to save it along with the TfidfVectorizer using pickle. This will allow us to load the model and vectorizer in the FastAPI app.\nimport pickle # Save the trained model with open(\u0026#39;model.pkl\u0026#39;, \u0026#39;wb\u0026#39;) as model_file: pickle.dump(model, model_file) # Save the TF-IDF vectorizer with open(\u0026#39;tfidf_vectorizer.pkl\u0026#39;, \u0026#39;wb\u0026#39;) as vec_file: pickle.dump(tf_idf_vectorizer, vec_file) print(\u0026#34;Model and vectorizer saved!\u0026#34;) After running the script this will save your model (model.pkl) and the vectorizer (tfidf_vectorizer.pkl) so they can be loaded in the FastAPI application.\nStep 2: Create the FastAPI App # Next, create a FastAPI app that will load the trained model and vectorizer, and provide an endpoint for users to submit news articles and receive predictions.\nDirectory structure:\nfake-news-api/ │ ├── main.py # FastAPI app ├── model.pkl # Saved PassiveAggressiveClassifier model ├── tfidf_vectorizer.pkl # Saved TfidfVectorizer ├── requirements.txt # Python dependencies ├── Procfile # Heroku process file └── runtime.txt # Specify Python version main.py:\nfrom fastapi import FastAPI from pydantic import BaseModel import pickle import numpy as np # Initialize FastAPI app app = FastAPI(title=\u0026#34;Fake News Detection API\u0026#34;, version=\u0026#34;1.0\u0026#34;) # Load the saved model and vectorizer with open(\u0026#34;model.pkl\u0026#34;, \u0026#34;rb\u0026#34;) as model_file: model = pickle.load(model_file) with open(\u0026#34;tfidf_vectorizer.pkl\u0026#34;, \u0026#34;rb\u0026#34;) as vec_file: tfidf_vectorizer = pickle.load(vec_file) # Define the request body structure using Pydantic class NewsArticle(BaseModel): text: str # Define the prediction endpoint @app.post(\u0026#34;/predict\u0026#34;) def predict_fake_news(article: NewsArticle): # Vectorize the incoming text vectorized_text = tfidf_vectorizer.transform([article.text]) # Make a prediction prediction = model.predict(vectorized_text) # Return the result result = \u0026#34;Fake\u0026#34; if prediction[0] == 0 else \u0026#34;True\u0026#34; return {\u0026#34;prediction\u0026#34;: result} # Home route @app.get(\u0026#34;/\u0026#34;) def read_root(): return {\u0026#34;message\u0026#34;: \u0026#34;Welcome to the Fake News Detection API! Visit /docs for Swagger documentation.\u0026#34;} Step 3: Prepare requirements.txt # You need to list all the dependencies your project needs in a requirements.txt file:\nfastapi uvicorn scikit-learn pandas numpy gunicorn Step 4: Run the application Locally # You can run the FastAPI application using Uvicorn. To do so, open your terminal or command prompt and navigate to the directory containing main.py. Then, run:\nuvicorn main:app --reload This will:\nStart the FastAPI server. Automatically reload the server if you make any changes to the code. Run the app at http://127.0.0.1:8000/. Open your browser and navigate to http://127.0.0.1:8000/. You should see the welcome message from your API. To explore and test the API, go to http://127.0.0.1:8000/docs. This will show an interactive Swagger UI where you can test the /predict endpoint by submitting a news article\u0026rsquo;s text. Testing the API # The API was tested using several news articles for both real and fake news categories. Here are a few examples:\nExample 1: Real News\nText: \u0026ldquo;The U.S. Congress passed a $1.9 trillion COVID-19 relief package on Wednesday, aimed at providing economic aid to millions of Americans impacted by the pandemic.\u0026rdquo; Prediction: True Example 2: Fake News\nText: \u0026ldquo;Aliens have landed on Earth and established a base in Antarctica. Government officials are working with them to develop advanced technology.\u0026rdquo; Prediction: Fake What to Do After Testing Locally # Once everything is working as expected on your local machine, you\u0026rsquo;re ready to push the app to Heroku for deployment.\nFor that, set up requirements.txt, Procfile, and push your code to Heroku.\nCreate the Procfile # Heroku needs a Procfile to understand how to run your application:\nweb: gunicorn -w 4 -k uvicorn.workers.UvicornWorker main:app -w 4: This specifies 4 worker processes for handling multiple requests. -k uvicorn.workers.UvicornWorker: Uses Uvicorn workers to handle the ASGI app. Specify the Python Version # Optionally, you can create a runtime.txt file to specify the Python version you want to use:\npython-3.9.12 Now, initialize Git and Push to Heroku.\nGratitude # This was the first problem involving integration with an external FastAPI application. I learned how to store the trained model and later use it for external applications. I also explored the deployment process. Looking forward to the next day\nStay Tuned!\n","date":"1 October 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-21/","section":"Challenges","summary":"It marks Day 21 of my 30 Days, 30 Machine Learning Projects Challenge. Today’s challenge was focused on deploying a machine learning model for real-time predictions. I chose to deploy my Fake News Detection model using FastAPI as the API framework and Heroku for hosting the app. Curious about how it went? Read on to see the results!","title":"Day 21 - Deploy a Machine Learning Model Using FastAPI and Heroku for Real-Time Predictions","type":"challenge"},{"content":"Hey, it is day 20 day of the 30 days 30 ML projects challenge. Here is the full code for your LDA topic modeling project using the New York Times Comments Dataset. It includes all steps: data preprocessing, LDA model building, and visualization.\nIf you want to see the code, you can find it here: GIT REPO.\nCode Flow # 1. Importing Libraries # import pandas as pd import re import nltk from nltk.corpus import stopwords from nltk.tokenize import word_tokenize from nltk.stem import WordNetLemmatizer from gensim import corpora import gensim import pyLDAvis.gensim Pandas: We use this to load and manipulate the dataset. nltk: The Natural Language Toolkit (NLTK) helps with text preprocessing (tokenization, stopwords, lemmatization). re: This is the regular expressions library in Python, which we use to remove unwanted characters (e.g., punctuation). gensim: A library for topic modeling that implements LDA, as well as word embeddings, text similarity, and more. pyLDAvis: A visualization tool specifically for LDA topic models, which provides an interactive display of the topics. 2. Loading the Dataset # df = pd.read_csv(\u0026#39;your_dataset.csv\u0026#39;) # Replace with the path to your dataset # Select the \u0026#39;snippet\u0026#39; column text_data = df[\u0026#39;snippet\u0026#39;].dropna().tolist() We load the New York Times Comments Dataset into a pandas DataFrame. This dataset has multiple columns, but we’re interested in the text-related columns. We use the snippet column, which contains short text snippets of the articles. This will be our source of text for topic modeling. 3. Preprocessing Setup # nltk.download(\u0026#39;stopwords\u0026#39;) nltk.download(\u0026#39;punkt\u0026#39;) nltk.download(\u0026#39;wordnet\u0026#39;) stop_words = set(stopwords.words(\u0026#39;english\u0026#39;)) lemmatizer = WordNetLemmatizer() Download NLTK Resources: We download NLTK\u0026rsquo;s stopwords, tokenization (punkt), and lemmatization resources (wordnet) so that we can preprocess the text. Stopwords: Words like \u0026ldquo;and,\u0026rdquo; \u0026ldquo;is,\u0026rdquo; \u0026ldquo;in,\u0026rdquo; etc., which don’t contribute much to the meaning of the text, are removed to improve the quality of the topic modeling. Lemmatizer: This reduces words to their base form. For example, \u0026ldquo;running\u0026rdquo; becomes \u0026ldquo;run.\u0026rdquo; Lemmatization helps group words with similar meanings. 4. Text Preprocessing # def preprocess(text): # Convert to lowercase text = text.lower() # Remove punctuation and numbers text = re.sub(r\u0026#39;\\W+\u0026#39;, \u0026#39; \u0026#39;, text) # Tokenize tokens = word_tokenize(text) # Remove stopwords and lemmatize tokens = [lemmatizer.lemmatize(word) for word in tokens if word not in stop_words] return tokens Convert to Lowercase: We convert everything to lowercase to avoid \u0026ldquo;Apple\u0026rdquo; and \u0026ldquo;apple\u0026rdquo; being treated as separate words. Remove Punctuation and Numbers: Using regular expressions, we remove unwanted characters like punctuation (. or ,) and numbers. Tokenization: Tokenizing breaks the text into individual words. Stopword Removal and Lemmatization: We remove stopwords and apply lemmatization to reduce words to their base form. cleaned_data = [preprocess(text) for text in text_data] We apply the preprocess function to each snippet of text and store the cleaned, tokenized data. 5. Creating Dictionary and Corpus # dictionary = corpora.Dictionary(cleaned_data) corpus = [dictionary.doc2bow(text) for text in cleaned_data] Dictionary: The dictionary maps each unique word in the dataset to a unique integer ID. This is necessary for Gensim\u0026rsquo;s LDA model to operate. Corpus: The corpus is a Bag-of-Words (BoW) representation of the text. It converts each document (snippet) into a list of tuples where each tuple represents the word’s ID and its count in the document. Example\nIf the cleaned text looks like this: ['apple', 'banana', 'apple'], the dictionary might map:\n\u0026ldquo;apple\u0026rdquo; → 0 \u0026ldquo;banana\u0026rdquo; → 1 The corpus for this snippet would be [(0, 2), (1, 1)], meaning \u0026ldquo;apple\u0026rdquo; appears twice and \u0026ldquo;banana\u0026rdquo; appears once. 6. Building the LDA Model # lda_model = gensim.models.ldamodel.LdaModel(corpus, num_topics=5, id2word=dictionary, passes=10) LDA Model: We create the LDA model using Gensim. corpus: The corpus (Bag-of-Words) representation of the cleaned data. num_topics=5: We ask LDA to find 5 topics. You can adjust this to any number of topics you expect. id2word=dictionary: This parameter maps the word IDs in the corpus back to actual words. passes=10: This specifies how many times the algorithm should pass over the entire corpus. More passes can lead to better topic distribution but will take longer. 7. Displaying Topics # topics = lda_model.print_topics(num_words=10) for topic in topics: print(topic) Print Topics: This displays the top 10 words in each of the 5 topics. LDA uses a probabilistic approach to assign words to topics, so each topic is represented by the words most likely to appear in that topic. Output Example:\n0: 0.025*\u0026#34;apple\u0026#34; + 0.018*\u0026#34;banana\u0026#34; + 0.015*\u0026#34;market\u0026#34; + ... 1: 0.021*\u0026#34;company\u0026#34; + 0.019*\u0026#34;technology\u0026#34; + 0.017*\u0026#34;innovation\u0026#34; + ... Interpretation: The output indicates that words like \u0026ldquo;apple,\u0026rdquo; \u0026ldquo;banana,\u0026rdquo; and \u0026ldquo;market\u0026rdquo; are prominent in Topic 0, while words like \u0026ldquo;company,\u0026rdquo; \u0026ldquo;technology,\u0026rdquo; and \u0026ldquo;innovation\u0026rdquo; are more frequent in Topic 1. 8. Visualizing Topics using pyLDAvis # pyLDAvis.enable_notebook() lda_vis = pyLDAvis.gensim.prepare(lda_model, corpus, dictionary) pyLDAvis.display(lda_vis) pyLDAvis: This tool provides an interactive visualization for topic models. It shows how the topics are distributed across documents and which words are strongly associated with each topic. Interactivity: You can explore each topic by clicking on it and seeing the most frequent words in that topic. Final Thoughts: # Adjust Number of Topics: Based on the coherence of the topics, you might want to adjust num_topics to a different number (e.g., 3, 7, 10). Preprocessing: If the results are not meaningful, consider improving the preprocessing step (e.g., adding more custom stopwords). Further Analysis: You can explore which articles are most strongly associated with each topic and gain more insights from the model. Gratitude # It was my first problem on NLTK. I did not perform good however i plan to solve good 10-15 problems on it in future to get better understanding of this topic.\nStay Tuned for day 21!\n","date":"30 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-20/","section":"Challenges","summary":"Today marks Day 20 of my 30 Days, 30 Machine Learning Projects Challenge. The problem is to Create a topic model using Latent Dirichlet Allocation (LDA). Curious about how it went? Read on to see the results!","title":"Day 20 - 30 Days 30 ML Projects: Create a topic model using Latent Dirichlet Allocation (LDA)","type":"challenge"},{"content":"Today’s challenge was about predicting customer churn, a crucial task for businesses that want to identify customers who are likely to cancel their subscriptions. We used an XGBoost model for this task, leveraging its powerful gradient boosting algorithm to achieve a high-performing classifier.\nIf you want to see the code, you can find it here: GIT REPO.\nUnderstanding the Data # We used a Telco Customer Churn dataset from Kaggle, which contains customer information such as gender, tenure, contract type, and monthly charges. The target variable is Churn, indicating whether the customer churned (Yes/No). Here\u0026rsquo;s a quick look at the data:\ncustomerID | gender | SeniorCitizen | Partner | Dependents | tenure | Contract | MonthlyCharges | Churn 7590-VHVEG | Female | 0 | Yes | No | 1 | Month-to-month | 29.85 | No Code Walkthrough # Let\u0026rsquo;s import the libraries first\nimport pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import LabelEncoder, OneHotEncoder from sklearn.compose import ColumnTransformer from xgboost import XGBClassifier from sklearn.metrics import accuracy_score, classification_report, confusion_matrix import matplotlib.pyplot as plt import seaborn as sns Step 1: Load the Data # data = pd.read_csv(\u0026#39;dataset/telco_customer_churn.csv\u0026#39;) We loaded the data and took a look at its structure.\nStep 2: Handle Missing Values # One of the columns, TotalCharges, had some missing values. We filled these values with the median for simplicity.\ndata[\u0026#39;TotalCharges\u0026#39;] = pd.to_numeric(data[\u0026#39;TotalCharges\u0026#39;], errors=\u0026#39;coerce\u0026#39;) data[\u0026#39;TotalCharges\u0026#39;].fillna(data[\u0026#39;TotalCharges\u0026#39;].median(), inplace=True) Step 3: Encode the Categorical Data # To prepare the categorical data, we used a hybrid encoding approach:\nLabel Encoding for binary features (like gender, Partner, Dependents). One-Hot Encoding for multi-category features (like Contract, PaymentMethod). # Label Encoding for binary columns label_enc = LabelEncoder() binary_cols = [\u0026#39;gender\u0026#39;, \u0026#39;Partner\u0026#39;, \u0026#39;Dependents\u0026#39;, \u0026#39;PhoneService\u0026#39;, \u0026#39;PaperlessBilling\u0026#39;, \u0026#39;Churn\u0026#39;] for col in binary_cols: data[col] = label_enc.fit_transform(data[col]) # One-Hot Encoding for multi-category columns multi_cat_cols = [\u0026#39;Contract\u0026#39;, \u0026#39;PaymentMethod\u0026#39;, \u0026#39;InternetService\u0026#39;] data_encoded = pd.get_dummies(data, columns=multi_cat_cols, drop_first=True) Step 4: Split the Data into Features and Target # X = data_encoded.drop(\u0026#39;Churn\u0026#39;, axis=1) y = data_encoded[\u0026#39;Churn\u0026#39;] We used Churn as the target and dropped it from the feature set.\nStep 5: Split the Data into Training and Validation Sets # We split the data into 80% training and 20% validation sets.\nfrom sklearn.model_selection import train_test_split X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) Step 6: Build and Train the XGBoost Model # We built an XGBoost model and trained it on the data.\nmodel = XGBClassifier(n_estimators=100, learning_rate=0.1, random_state=42) model.fit(X_train, y_train) Step 7: Evaluate the Model # After training, we made predictions on the validation set and evaluated the model’s performance using accuracy, a confusion matrix, and a classification report.\npredictions = model.predict(X_val) accuracy = accuracy_score(y_val, predictions) conf_matrix = confusion_matrix(y_val, predictions) class_report = classification_report(y_val, predictions) print(f\u0026#34;Accuracy: {accuracy}\u0026#34;) print(f\u0026#34;Confusion Matrix:\\n{conf_matrix}\u0026#34;) print(f\u0026#34;Classification Report:\\n{class_report}\u0026#34;) Model Performance # The model achieved an accuracy of 100%, which is solid for a churn prediction model.\nAccuracy Score: 1.0 Confusion Matrix: [[1036 0] [ 0 373]] Classification Report: precision recall f1-score support 0 1.00 1.00 1.00 1036 1 1.00 1.00 1.00 373 accuracy 1.00 1409 macro avg 1.00 1.00 1.00 1409 weighted avg 1.00 1.00 1.00 1409 Gratitude # This challenge deepened my understanding of handling customer churn data and using XGBoost for classification tasks. It was great exploring different encoding techniques to preprocess categorical data efficiently.\nStay Tuned!\n","date":"29 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-19/","section":"Challenges","summary":"Today marks Day 19 of my 30 Days, 30 Machine Learning Projects Challenge. The problem is to Predict customer churn using XGBoost. Curious about how it went? Read on to see the results!","title":"Day 19 - 30 Days 30 ML Projects: Customer Churn Prediction with XGBoost","type":"challenge"},{"content":"On Day 18 of the 30 Days 30 Machine Learning Projects Challenge, the task was to predict stock prices using the ARIMA model. ARIMA (Auto-Regressive Integrated Moving Average) is one of the most widely used techniques for time series forecasting, especially for data that shows trends or seasonality.\nIf you want to see the code, you can find it here: GIT REPO.\nUnderstanding the Data # We used the MAANG Historical Stock Market Dataset and worked specifically with Apple stock prices. The dataset contains various columns, but for this project, we used the Close price, which represents the final trading price of the stock on each day.\nCode Workflow # Below is the step-by-step approach followed for solving this problem.\nStep 1: Load the Data and Preprocess # We\u0026rsquo;ll load the Apple stock dataset and focus on the \u0026lsquo;Date\u0026rsquo; and \u0026lsquo;Close\u0026rsquo; columns to predict future stock prices. Since ARIMA requires a continuous time series, we\u0026rsquo;ll set the \u0026lsquo;Date\u0026rsquo; column as the index.\nimport pandas as pd import matplotlib.pyplot as plt from statsmodels.tsa.arima.model import ARIMA from pandas.plotting import autocorrelation_plot import warnings warnings.filterwarnings(\u0026#34;ignore\u0026#34;) # Load the data data = pd.read_csv(\u0026#39;dataset/Apple.csv\u0026#39;) # Convert the \u0026#39;Date\u0026#39; column to datetime format and set it as the index data[\u0026#39;Date\u0026#39;] = pd.to_datetime(data[\u0026#39;Date\u0026#39;]) data.set_index(\u0026#39;Date\u0026#39;, inplace=True) # Plot the closing price to visualize the time series plt.figure(figsize=(10,6)) plt.plot(data[\u0026#39;Close\u0026#39;], label=\u0026#39;Apple Stock Closing Price\u0026#39;) plt.title(\u0026#39;Apple Stock Closing Price Over Time\u0026#39;) plt.xlabel(\u0026#39;Date\u0026#39;) plt.ylabel(\u0026#39;Close Price\u0026#39;) plt.legend() plt.show() Step 2: Check for Stationarity # For ARIMA to work well, we need a stationary time series. We\u0026rsquo;ll check for stationarity using a rolling mean and standard deviation.\n# Calculate rolling statistics to check for stationarity rolling_mean = data[\u0026#39;Close\u0026#39;].rolling(window=12).mean() rolling_std = data[\u0026#39;Close\u0026#39;].rolling(window=12).std() # Plot rolling statistics plt.figure(figsize=(10,6)) plt.plot(data[\u0026#39;Close\u0026#39;], color=\u0026#39;blue\u0026#39;, label=\u0026#39;Original Close Price\u0026#39;) plt.plot(rolling_mean, color=\u0026#39;red\u0026#39;, label=\u0026#39;Rolling Mean\u0026#39;) plt.plot(rolling_std, color=\u0026#39;black\u0026#39;, label=\u0026#39;Rolling Std\u0026#39;) plt.title(\u0026#39;Rolling Mean \u0026amp; Standard Deviation for Stationarity Check\u0026#39;) plt.legend() plt.show() Step 3: Differencing the Data to Make it Stationary # If the data is not stationary, we’ll apply differencing to remove trends and seasonality.\n# Differencing the data to make it stationary data_diff = data[\u0026#39;Close\u0026#39;].diff().dropna() # Plot the differenced data plt.figure(figsize=(10,6)) plt.plot(data_diff, label=\u0026#39;Differenced Data\u0026#39;) plt.title(\u0026#39;Differenced Time Series Data\u0026#39;) plt.legend() plt.show() Step 4: Fit the ARIMA Model # Now that we have stationary data, we can fit an ARIMA model to it. We\u0026rsquo;ll use ARIMA\u0026rsquo;s parameters (p, d, q) to control the autoregression (AR), differencing (I), and moving average (MA) parts.\n# Fit the ARIMA model model = ARIMA(data[\u0026#39;Close\u0026#39;], order=(5, 1, 0)) # You can experiment with other (p, d, q) values model_fit = model.fit() # Summary of the model print(model_fit.summary()) let\u0026rsquo;s decode the parameters (p=5, d=1, q=0):\np (Auto-Regressive part): Looks at the number of lag observations included in the model. d (Differencing part): Indicates how many times the data needs to be differenced to make it stationary. q (Moving Average part): Determines the size of the moving average window.\nStep 5: Make Predictions # We forecasted the next 30 days of Apple stock prices and plotted the predictions against the actual prices.\n# Forecast future prices forecast = model_fit.forecast(steps=30) # Forecast for 30 days ahead plt.figure(figsize=(10,6)) plt.plot(data[\u0026#39;Close\u0026#39;], label=\u0026#39;Actual Prices\u0026#39;) plt.plot(forecast, label=\u0026#39;Predicted Prices\u0026#39;, color=\u0026#39;red\u0026#39;) plt.title(\u0026#39;Apple Stock Price Prediction with ARIMA\u0026#39;) plt.xlabel(\u0026#39;Date\u0026#39;) plt.ylabel(\u0026#39;Close Price\u0026#39;) plt.legend() plt.show() The ARIMA model was able to predict short-term future values for Apple stock prices.\nGratitude # It was really exciting working with ARIMA to predict stock prices! Looking forward to tomorrow’s challenge!\nStay Tuned!\n","date":"28 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-18/","section":"Challenges","summary":"On Day 18, we used the ARIMA model to predict stock prices based on historical closing prices. By using the ARIMA (Auto-Regressive Integrated Moving Average) approach, we built a model that can predict short-term future values in the time series.","title":"Day 18 - 30 Days 30 ML Projects: Time Series Forecasting of Stock Prices with ARIMA Model","type":"challenge"},{"content":"On Day 17 of the 30 Days 30 Machine Learning Projects Challenge, the task was to predict whether a person would develop diabetes based on various medical factors such as glucose levels, insulin levels, and age. This is a binary classification problem where the goal is to predict if a person is diabetic (1) or not (0).\nIf you want to see the code, you can find it here: GIT REPO.\nUnderstanding the Data # We used the Pima Indians Diabetes Dataset, which includes medical records of women aged 21 and above. The dataset contains various features related to pregnancies, glucose levels, blood pressure, skin thickness, and more. Here\u0026rsquo;s a glimpse of the data:\nPregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigreeFunction Age Outcome 0 6 148 72 35 0 33.6 0.627 50 1 1 1 85 66 29 0 26.6 0.351 31 0 2 8 183 64 0 0 23.3 0.672 32 1 3 1 89 66 23 94 28.1 0.167 21 0 4 0 137 40 35 168 43.1 2.288 33 1 Outcome:\n1 means the patient is diabetic. 0 means the patient is not diabetic. Code Workflow # Here’s the step-by-step breakdown of how I approached this problem:\nStep 1: Load the Data # First, I loaded the dataset using Pandas to explore and understand the data.\nimport pandas as pd # Load the dataset data = pd.read_csv(\u0026#39;dataset/diabetes.csv\u0026#39;) print(data.head()) Step 2: Preprocess the Data # Next, I separated the features (X) from the target (y).\nX = data.drop(\u0026#39;Outcome\u0026#39;, axis=1) # Features y = data[\u0026#39;Outcome\u0026#39;] # Target Step 3: Split the Data # The data was then split into training and validation sets with an 80-20 split.\nfrom sklearn.model_selection import train_test_split X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) Step 4: Build and Train the Models # Decision Tree: I trained a Decision Tree Classifier as the first model.\nfrom sklearn.tree import DecisionTreeClassifier # Build and train the Decision Tree model decision_tree = DecisionTreeClassifier(random_state=42) decision_tree.fit(X_train, y_train) Random Forest: Next, I trained a Random Forest Classifier with 100 trees and specific parameters.\nfrom sklearn.ensemble import RandomForestClassifier # Build and train the Random Forest model random_forest = RandomForestClassifier(n_estimators=100, min_samples_leaf=1, min_samples_split=5, random_state=42) random_forest.fit(X_train, y_train) Step 5: Make Predictions and Evaluate # After training, I made predictions on the validation set and evaluated both models using accuracy, confusion matrices, and classification reports.\nfrom sklearn.metrics import accuracy_score, confusion_matrix, classification_report # Decision Tree dt_predictions = decision_tree.predict(X_val) dt_accuracy = accuracy_score(y_val, dt_predictions) dt_confusion_matrix = confusion_matrix(y_val, dt_predictions) dt_classification_report = classification_report(y_val, dt_predictions) print(f\u0026#34;Decision Tree Accuracy Score: {dt_accuracy}\u0026#34;) print(f\u0026#34;Decision Tree Confusion Matrix: {dt_confusion_matrix}\u0026#34;) print(f\u0026#34;Decision Tree Classification Report: {dt_classification_report}\u0026#34;) # Random Forest rf_predictions = random_forest.predict(X_val) rf_accuracy = accuracy_score(y_val, rf_predictions) rf_confusion_matrix = confusion_matrix(y_val, rf_predictions) rf_classification_report = classification_report(y_val, rf_predictions) print(f\u0026#34;Random Forest Accuracy Score: {rf_accuracy}\u0026#34;) print(f\u0026#34;Random Forest Confusion Matrix: {rf_confusion_matrix}\u0026#34;) print(f\u0026#34;Random Forest Classification Report: {rf_classification_report}\u0026#34;) Results:\nDecision Tree Accuracy: 74% Random Forest Accuracy: 73% Step 6: Visualization # I visualized the confusion matrices for both models using Seaborn heatmaps.\nimport matplotlib.pyplot as plt import seaborn as sns # Decision Tree plt.figure(figsize=(7, 5)) sns.heatmap(dt_confusion_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=[\u0026#39;No Diabetes\u0026#39;, \u0026#39;Diabetes\u0026#39;], yticklabels=[\u0026#39;No Diabetes\u0026#39;, \u0026#39;Diabetes\u0026#39;]) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Decision Tree Confusion Matrix\u0026#39;) plt.show() # Random Forest plt.figure(figsize=(7, 5)) sns.heatmap(rf_confusion_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=[\u0026#39;No Diabetes\u0026#39;, \u0026#39;Diabetes\u0026#39;], yticklabels=[\u0026#39;No Diabetes\u0026#39;, \u0026#39;Diabetes\u0026#39;]) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Random Forest Confusion Matrix\u0026#39;) plt.show() Model Performance # The Decision Tree performed slightly better with a 74% accuracy score, while the Random Forest model performed at 73% accuracy. I tried improving the Random Forest model by performing hyperparameter tuning.\nfrom sklearn.model_selection import GridSearchCV param_grid = { \u0026#39;n_estimators\u0026#39;: [100, 200], \u0026#39;max_depth\u0026#39;: [None, 10, 20], \u0026#39;min_samples_split\u0026#39;: [2, 5], \u0026#39;min_samples_leaf\u0026#39;: [1, 2] } grid_search = GridSearchCV(estimator=rf, param_grid=param_grid, cv=5, n_jobs=-1, verbose=2) grid_search.fit(X_train, y_train) # Display the best parameters and score print(\u0026#34;Best Parameters:\u0026#34;, grid_search.best_params_) print(\u0026#34;Best Score:\u0026#34;, grid_search.best_score_) After tuning, the best parameters were:\nBest Parameters: {\u0026#39;max_depth\u0026#39;: None, \u0026#39;min_samples_leaf\u0026#39;: 1, \u0026#39;min_samples_split\u0026#39;: 5, \u0026#39;n_estimators\u0026#39;: 100} Best Score: 0.783 However, the model\u0026rsquo;s accuracy remained consistent at 73%.\nGratitude # This project was a deep dive into comparing Decision Trees and Random Forests. I learned a lot about tuning models and how important it is to understand the trade-offs between complexity and performance.\nStay tuned for Day 18!\n","date":"27 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-17/","section":"Challenges","summary":"Hey, It is Day 17 of the 30 Days 30 Machine Learning Project Challenge, and today we predicted diabetes onset using Decision Trees and Random Forests. Curious about how it went? Read on to see the results!","title":"Day 17 - 30 Days 30 ML Projects: Predict Diabetes Onset Using Decision Trees and Random Forests","type":"challenge"},{"content":"Today, I tackled a real-time face detection problem using OpenCV. The goal was to implement a system that could detect faces in real-time from a webcam feed and highlight them with bounding boxes.\nIf you want to see the code, you can find it here: GIT REPO.\nThe Solution # We used OpenCV, a powerful computer vision library, to process video streams and detect faces. The Haar Cascade Classifier was the key tool for recognizing face patterns. It works by scanning the image for areas that look like a human face, then drawing a rectangle around them.\nCode Workflow # Lets import the required libraries first:\nimport cv2 This imports the OpenCV library. OpenCV (Open Source Computer Vision Library) is a library of programming functions primarily aimed at real-time computer vision.\nStep 1: Load the Pre-trained Haar Cascade for Face Detection # face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + \u0026#39;haarcascade_frontalface_default.xml\u0026#39;) CascadeClassifier: This function loads the pre-trained Haar Cascade XML file which contains the model for detecting faces. OpenCV comes with several pre-trained models for face detection, and here we are using the \u0026lsquo;haarcascade_frontalface_default.xml\u0026rsquo; model. cv2.data.haarcascades: This points to the directory where OpenCV stores its pre-trained models, so the cascade file can be located easily. haarcascade_frontalface_default.xml: This XML file contains the Haar Cascade data for detecting human frontal faces. Step 2: Capture Video from the Webcam # cap = cv2.VideoCapture(0) cv2.VideoCapture(0): This creates an object cap which starts accessing the webcam. The parameter (0) specifies that the default webcam (if you have more than one camera, 1, 2, etc. will refer to other cameras). The webcam is now ready to capture frames in real-time. Step 3: Process Each Frame from the Webcam Feed # while True: ret, frame = cap.read() # Read each frame if not ret: break while True: This starts an infinite loop that continuously processes the video stream frame by frame. ret, frame = cap.read(): This reads the current frame from the webcam. The read() method returns two values: ret: A boolean that indicates whether the frame was successfully captured (True) or not (False). frame: The captured frame (an image represented as a NumPy array). if not ret: If ret is False, it means there was an issue capturing the frame (e.g., camera disconnected), so we break the loop. Convert Frame to Grayscale # gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY): This converts the captured frame from a colored image (BGR) to grayscale. Face detection using Haar Cascades performs better on grayscale images because they are computationally less complex compared to color images. Step 4: Detect Faces in the Frame # faces = face_cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=5, minSize=(30, 30)) detectMultiScale: This method detects objects (in our case, faces) in the grayscale image. It returns a list of rectangles where faces are detected.\ngray: The grayscale image where the detection will take place.\nscaleFactor=1.1: Specifies how much the image size is reduced at each scale. 1.1 means the image is reduced by 10% at each scale, helping to detect faces at different sizes.\nminNeighbors=5: Specifies how many neighbors each rectangle candidate should have to retain it. Higher values result in fewer detections but with higher quality.\nminSize=(30, 30): The minimum possible size of the detected face. Any face smaller than 30x30 pixels will not be considered.\nDraw Rectangles Around Detected Faces # for (x, y, w, h) in faces: cv2.rectangle(frame, (x, y), (x+w, y+h), (255, 0, 0), 2) for (x, y, w, h) in faces: Loops through all the detected faces. Each face is represented by a rectangle with:\n(x, y): Coordinates of the top-left corner of the rectangle. (w, h): The width and height of the rectangle. cv2.rectangle(frame, (x, y), (x+w, y+h), (255, 0, 0), 2):\nDraws a rectangle on the original colored frame. (255, 0, 0): Specifies the rectangle color in BGR (Blue in this case). 2: The thickness of the rectangle. Display the Frame with Detected Faces # cv2.imshow(\u0026#39;Face Detection\u0026#39;, frame) cv2.imshow: This function displays the current frame in a window titled \u0026lsquo;Face Detection\u0026rsquo;. The frame now contains rectangles drawn around detected faces. Exit the Loop and Close the Webcam # if cv2.waitKey(1) \u0026amp; 0xFF == ord(\u0026#39;q\u0026#39;): break cv2.waitKey(1): This function waits for a key press for 1 millisecond. The 1 millisecond delay is needed to give OpenCV time to refresh the window showing the video. \u0026amp; 0xFF: Ensures compatibility across different operating systems. ord('q'): This checks if the \u0026lsquo;q\u0026rsquo; key was pressed. If it was, the loop breaks, and the program ends. Step 5: Release the Resources and Close All Windows # cap.release() cv2.destroyAllWindows() cap.release(): This releases the webcam so other applications can access it. cv2.destroyAllWindows(): Closes all the OpenCV windows that were opened during the program\u0026rsquo;s execution. Output # When the script runs, the webcam feed opens, and the faces detected are highlighted with blue rectangles.\nModel Performance # Since this project didn’t involve a trained model, there were no accuracy metrics to track. However, the detection worked smoothly for faces in good lighting and straightforward angles.\nGratitude # This project was my first venture into OpenCV, and I’m really excited about the possibilities of computer vision. I’m eager to explore more in this space and apply it to other projects!\nStay Tuned!\n","date":"26 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-16/","section":"Challenges","summary":"Day 16 of the challenge, real-time face detection in a webcam feed using OpenCV, was my first OpenCV problem, and I had a blast working with computer vision. This project opens up so many possibilities for real-time applications!","title":"Day 16 - 30 Days 30 ML Projects: Real-time Face Detection in a Webcam Feed using OpenCV","type":"challenge"},{"content":"Today’s challenge was to predict house prices using the Ames Housing dataset, a more comprehensive and detailed dataset compared to the simpler California Housing dataset. The task was to build a regression model using XGBoost, a high-performance implementation of gradient boosted decision trees.\nIf you want to see the code, you can find it here: GIT REPO.\nUnderstanding the Data # We used the Ames Housing Dataset, which contains 80 features describing various aspects of residential properties in Ames, Iowa. These features range from the lot area, year built, and overall quality to more intricate details like basement quality and garage condition.\nThe target variable in this problem is the SalePrice, representing the selling price of the houses.\nCode Workflow # Load the Data Preprocess the Data (Handling Missing Values and Categorical Data) Split the Data Train the XGBoost Model Evaluate the Model Visualize Feature Importance Let’s break down each step.\nStep 1: Load the Data # We started by loading the Ames Housing dataset from Kaggle into a pandas DataFrame for easy manipulation and exploration.\nimport pandas as pd # Load dataset data = pd.read_csv(\u0026#39;dataset/AmesHousing.csv\u0026#39;) print(data.head()) The dataset contains both categorical and numerical columns, which required different preprocessing approaches.\nStep 2: Preprocess the Data # In this step, I handled the missing values and encoded the categorical features. For missing numerical values, I filled them with the median of the respective column, while for categorical data, I filled the missing values with \u0026lsquo;None\u0026rsquo;.\nI used One Hot Encoding to convert categorical features into numerical ones.\n# Handle missing values and encode categorical data for col in data.columns: if data[col].dtypes in [np.int64, np.float64]: data[col].fillna(data[col].median(), inplace=True) else: data[col].fillna(\u0026#39;None\u0026#39;, inplace=True) # One Hot Encoding for categorical features data_encoded = pd.get_dummies(data, drop_first=True) # Define feature and target sets X = data_encoded.drop(\u0026#39;SalePrice\u0026#39;, axis=1) y = data_encoded[\u0026#39;SalePrice\u0026#39;] Step 3: Split the Data # We split the data into 80% for training and 20% for validation.\nfrom sklearn.model_selection import train_test_split # Split the data into training and validation sets X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) Step 4: Train the Model # For this problem, I used XGBoost Regressor with 100 estimators and a learning rate of 0.1. XGBoost is ideal for handling large datasets with a mix of numerical and categorical features, offering high performance and accuracy.\nfrom xgboost import XGBRegressor # Build and train the model model = XGBRegressor(n_estimators=100, learning_rate=0.1, random_state=42) model.fit(X_train, y_train) Step 5: Make Predictions and Evaluate the Model # After training the model, I made predictions on the validation set and calculated the Root Mean Squared Error (RMSE), which helps measure the average error in predicting house prices.\nfrom sklearn.metrics import mean_squared_error import numpy as np # Make predictions and evaluate the model predictions = model.predict(X_val) mse = mean_squared_error(y_val, predictions) rmse = np.sqrt(mse) print(\u0026#34;Root Mean Square Error: \u0026#34;, rmse) Step 6: Visualization # XGBoost has a built-in function for plotting feature importance, which helps understand which features contribute the most to the predictions.\nimport matplotlib.pyplot as plt import xgboost # Plot feature importance plt.figure(figsize=(7, 5)) xgboost.plot_importance(model, max_num_features=10) plt.title(\u0026#39;Top 10 Important Features\u0026#39;) plt.show() Model Performance # The model performed well, achieving a root mean square error (RMSE) of 24,059.40.\nGratitude # This is the first day of the 3rd week. We explored an advanced model for the same problem we solved on Day 1. I\u0026rsquo;m excited to continue and complete this challenge.\nStay tuned!\n","date":"25 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-15/","section":"Challenges","summary":"Day 15 of the 30 Days Machine Learning Challenge involves using XGBoost to predict house prices. The Ames Housing dataset gives a rich feature set, making it ideal for regression tasks. This project steps through data preprocessing, model training, evaluation, and visualization of feature importance using XGBoost.","title":"Day 15 - 30 Days 30 ML Projects: Predict House Prices with XGBoost","type":"challenge"},{"content":"The task for Day 14 is to use K-Means clustering to segment grocery store customers based on their purchasing history. Clustering helps businesses identify customer groups with similar buying habits, making it easier to create targeted marketing strategies and personalized customer experiences.\nIf you want to see the code, you can find it here: GIT REPO.\nUnderstanding the Data # We used the Wholesale Customers Dataset from UCI’s Machine Learning Repository. The dataset includes features like:\nFresh: Annual spending on fresh products (fruit, vegetables, etc.) Milk: Annual spending on milk products Grocery: Annual spending on groceries Frozen: Annual spending on frozen products Detergents_Paper: Annual spending on detergents and paper Delicatessen: Annual spending on delicatessen products We used these features to cluster customers into different segments based on their purchasing behavior.\nDownload and place it in dataset directory of your project.\nCode Workflow # Here’s the step-by-step process followed:\nLoad the Data Preprocess the Data Apply K-Means Clustering Use the Elbow Method to Find the Optimal K Visualize the Clusters with PCA Step 1: Load the Data # We started by loading the Wholesale Customers Dataset into a pandas DataFrame:\nimport pandas as pd data = pd.read_csv(\u0026#39;dataset/wholesale_customers_data.csv\u0026#39;) print(data.head()) Step 2: Preprocess the Data # We checked for missing values and then scaled the data to ensure all features are on the same scale. K-Means is sensitive to large variances, so we applied StandardScaler to normalize the data:\nfrom sklearn.preprocessing import StandardScaler scaler = StandardScaler() data_scaled = scaler.fit_transform(data) Step 3: Apply the K-Means Algorithm # To determine the optimal number of clusters (K), we used the Elbow Method. This method looks at the sum of squared distances from each point to its assigned cluster center (inertia) and plots it against various values of K. The \u0026ldquo;elbow\u0026rdquo; of the curve is the optimal number of clusters.\nfrom sklearn.cluster import KMeans import matplotlib.pyplot as plt sum_sq_dist_pt = [] # Sum of squared distances of each K for k in range(1, 11): kmeans = KMeans(n_clusters=k, random_state=42, n_init=\u0026#39;auto\u0026#39;) kmeans.fit(data_scaled) sum_sq_dist_pt.append(kmeans.inertia_) # Plot the Elbow curve plt.figure(figsize=(7, 5)) plt.plot(range(1, 11), sum_sq_dist_pt, marker=\u0026#39;o\u0026#39;) plt.xlabel(\u0026#39;Number of Clusters (K)\u0026#39;) plt.ylabel(\u0026#39;Inertia\u0026#39;) plt.title(\u0026#39;Elbow Method for Optimal K\u0026#39;) plt.show() From the elbow plot, we observed that K=3 was a good choice.\nStep 4: Train the Model # We selected K=3 based on the Elbow Method and retrained the K-Means algorithm on the scaled data:\nk_optimal = 3 kmeans = KMeans(n_clusters=k_optimal, random_state=42, n_init=\u0026#39;auto\u0026#39;) kmeans.fit(data_scaled) # Add the cluster label to the original dataset data[\u0026#39;cluster\u0026#39;] = kmeans.labels_ print(data.head()) Step 5: Visualize the Clusters Using PCA # Using Principal Component Analysis (PCA), we reduced the high-dimensional data to two principal components for visualization:\nfrom sklearn.decomposition import PCA pca = PCA(n_components=2) data_pca = pca.fit_transform(data_scaled) plt.figure(figsize=(10,7)) plt.scatter(data_pca[:, 0], data_pca[:, 1], c=kmeans.labels_, cmap=\u0026#39;viridis\u0026#39;) plt.xlabel(\u0026#39;Principal Component 1\u0026#39;) plt.ylabel(\u0026#39;Principal Component 2\u0026#39;) plt.title(\u0026#39;K-Means Clustering with PCA\u0026#39;) plt.show() Model Performance # Using K-Means Clustering and PCA visualization, we successfully segmented grocery store customers into 3 distinct clusters based on their purchase behavior. Each cluster represents a unique group of customers with similar spending patterns, which can be useful for targeted marketing or customer service strategies.\nGratitude # This project was a great introduction to unsupervised learning with K-Means and using the Elbow Method to find the optimal number of clusters. Learning how to visualize high-dimensional data with PCA also deepened my understanding of data representation. Looking forward to Day 15!\nStay tuned!\n","date":"24 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-14/","section":"Challenges","summary":"In Day 14 of the 30 Days 30 Machine Learning Projects Challenge, I worked on clustering grocery store customers using the K-Means algorithm. This task helps businesses better understand their customer segments, allowing for tailored marketing strategies based on purchase patterns.","title":"Day 14 - 30 Days 30 ML Projects: Cluster Grocery Store Customers with K-Means","type":"challenge"},{"content":" What is Classification Report: # Precision\nWhat it means: Out of all the predictions your model made for a specific class (e.g., all the times it predicted \u0026ldquo;jazz\u0026rdquo;), precision tells you how many were actually correct. Example: If the model predicted \u0026ldquo;jazz\u0026rdquo; 10 times but only 6 of those were correct, precision would be 6 /10 = 0.6 (or 60%). Recall\nWhat it means: Out of all the actual instances of a class in your dataset (e.g., how many times \u0026ldquo;jazz\u0026rdquo; really appears), recall tells you how many your model correctly identified. Example: If there are 8 \u0026ldquo;jazz\u0026rdquo; tracks, and your model correctly predicted 6 of them, recall would be 6 / 8 = 0.75 (or 75%). F1-Score\nWhat it means: The F1-score is the harmonic mean of precision and recall. It’s a single metric that balances both. If you want to focus equally on precision and recall, F1-score gives you a better picture. Example: If precision is 60% and recall is 75%, the F1-score would be the combination of the two: F1 = 2 * ( (Precision * Recall) / (Precision + Recall) ) = ( 2 X (0.6 * 0.75) / (0.6 + 0.75) ) = 0.67 Support\nWhat it means: Support simply tells you how many actual instances of each class there are in the test data. It helps you see if your dataset is balanced or if some classes have more examples than others. Example: Let’s look at a classification report snippet:\nprecision recall f1-score support blues 0.59 0.77 0.67 22 classical 0.90 0.93 0.91 28 country 0.59 0.59 0.59 22 Blues Precision: Out of all the times the model predicted \u0026ldquo;blues,\u0026rdquo; 59% were correct. Blues Recall: Out of all the actual \u0026ldquo;blues\u0026rdquo; songs, the model correctly identified 77%. Blues F1-Score: The F1-score balances precision and recall, in this case, 67%. Support: There were 22 actual \u0026ldquo;blues\u0026rdquo; tracks in the validation set. How It Helps: # Precision is important when false positives (wrongly classifying something as a genre) are costly. For example, if you don’t want a \u0026ldquo;classical\u0026rdquo; song mistakenly predicted as \u0026ldquo;hip-hop,\u0026rdquo; focus on precision. Recall is important when false negatives (missing instances of a genre) matter. For example, if it’s essential to catch all instances of \u0026ldquo;hip-hop,\u0026rdquo; recall is critical. The classification report helps you assess how well your model handles each class and where it struggles. You can also use it to compare models or fine-tune them.\nUnsupervised Learning # In unsupervised learning, the machine is given a dataset that doesn’t have any labeled output. The goal is for the algorithm to find hidden patterns or relationships within the data on its own, without being told what the “right answer” is.\nExample: Imagine you have a basket of mixed fruits, but you don’t know what types they are. An unsupervised learning algorithm would group similar fruits together based on features like size, color, and texture without knowing in advance which fruits are apples, oranges, etc.\nApplications:\nCustomer segmentation (grouping customers based on buying habits) Anomaly detection (finding unusual patterns) Data compression (dimensionality reduction) K-Means Clustering # K-Means is a popular unsupervised learning algorithm used for clustering. Its purpose is to divide data points into K clusters, where each cluster contains similar data points.\nHow it Works:\nChoosing K: You start by deciding how many clusters (K) you want to divide your data into.\nAssigning Cluster Centers: The algorithm randomly selects K points in your dataset as initial cluster centers (centroids).\nAssigning Points to Clusters: Each data point is assigned to the nearest centroid based on the distance (usually Euclidean distance). Points that are closer to a centroid are grouped into that cluster.\nRecalculating Centroids: After all points are assigned, the algorithm recalculates the centroids of the clusters by finding the average of all points in each cluster.\nRepeat: Steps 3 and 4 are repeated until the cluster assignments don’t change anymore (convergence).\nExample: Let’s say you have a dataset of customers, and each customer has two features: total amount spent and frequency of visits. If you set K to 3, K-Means might group the customers into three clusters: high spenders who visit often, low spenders who visit rarely, and those in between.\nElbow Method # The Elbow Method helps determine the optimal number of clusters (K) in K-Means.\nHow it Works:\nRun K-Means for different values of K (e.g., K=1, 2, 3, 4, etc.).\nFor each value of K, calculate the sum of squared distances (inertia) between data points and their assigned cluster centers. This tells you how tightly grouped your data points are within each cluster.\nPlot the Inertia against the number of clusters (K). The graph will usually have a bend or elbow.\nOptimal K: The point where the curve bends (the \u0026ldquo;elbow\u0026rdquo;) is considered the optimal K. Beyond this point, adding more clusters doesn’t improve the clustering significantly.\nExample: Imagine you’re trying to segment customers into groups based on their purchasing patterns. By using the elbow method, you might find that the ideal number of clusters is 3, as the graph bends at K=3. Going beyond 3 clusters wouldn’t add much extra value.\nARIMA # ARIMA Model stands for AutoRegressive Integrated Moving Average. It is used to forecast future values in a time series, like predicting stock prices over time. Here\u0026rsquo;s a simplified version:\nKey Concepts: # AutoRegressive (AR): This means that the model uses past values (previous stock prices) to predict future values. Think of it like this: if we know the stock price for the last few days, we might use those to predict what today’s price might be.\nExample: If you knew that the stock price was $100 on Monday, $102 on Tuesday, and $104 on Wednesday, you might predict that it will increase similarly on Thursday.\nIntegrated (I): Sometimes, data might be moving up or down over time (like a trend). To make predictions easier, the ARIMA model removes that trend, making the data more \u0026ldquo;stationary\u0026rdquo; (flat). It does this by looking at the difference between one day\u0026rsquo;s price and the previous day’s price.\nExample: If the stock price increases by $2 every day, the integrated part will take out that $2 jump so that it’s easier to see patterns in the changes.\nMoving Average (MA): This part looks at the errors in past predictions. It tries to correct for those errors by considering the difference between predicted prices and actual prices in the past. So, if the model predicted wrong a few days ago, it adjusts itself for better predictions now.\nExample: If you predicted that the stock would rise by $2 yesterday, but it only rose by $1, the model will use that mistake (error) to improve today’s prediction.\nAn Example: # Let’s say we want to predict the future price of a stock. Here’s how the ARIMA model would approach it:\nAR (AutoRegressive): It looks at past prices like:\n$100 on Monday $102 on Tuesday $104 on Wednesday It predicts that the price will be around $106 on Thursday, because it has been increasing by $2 each day.\nI (Integrated): It looks at the changes in prices:\nMonday to Tuesday: +$2 Tuesday to Wednesday: +$2 It calculates the differences and makes the data stationary (removing any trend).\nMA (Moving Average): It checks how good its past predictions were and uses that info:\nIf it predicted $106 but the price was $105, it learns from the mistake and adjusts its future predictions.\nTogether, the ARIMA model combines these three steps to make a more accurate prediction for stock prices (or any time-series data).\nStandard Scaling # Scaling is the process of transforming your data so that all features (variables) are on a similar scale or range. It’s commonly done in machine learning to ensure that no feature dominates the others simply because of its larger numerical range.\nMore technically, StandardScaler ensures that all features contribute equally by transforming the data to have a mean of 0 and a standard deviation of 1.\nLet’s understand scaling with an example:\nSuppose we have a small dataset with two features: height and weight. The values for these features are in different scales. Height has a larger numerical range than weight.\nHeight (cm): 160, 170, 150, 180, 175 Weight (kg): 65, 70, 55, 85, 75 Before Scaling # Let’s calculate the mean and standard deviation for each feature:\nHeight:\nMean: 167 Standard Deviation: 11.18 Weight:\nMean: 70 Standard Deviation: 10 After Applying StandardScaler: # For each value, we use the formula:\nScaled Value = (Original Value - Mean) / Standard Deviation\nFor example,\nFor height 160, scaled value will be (160 - 167) / 11.18 ~ -0.63 For weight 65, scaled value will be (65 - 70) / 10 ~ -0.5 Here’s how the scaled values would look:\nHeights: -0.63, 0.27, -1.52, 1.16, 0.72 Weights: -0.5, 0, -1.5, 1.5, 0.5 Visualizing the Output # Original Data:\nHeight ranges from 150 to 180 cm. Weight ranges from 55 to 85 kg. After Scaling:\nThe transformed height and weight values are now centered around 0, and their standard deviations are 1. This ensures that the data has zero mean and unit variance, meaning all features are on the same scale. Sequential Model # What Is a Sequential Model? In the world of deep learning, a Sequential model is like stacking building blocks one by one. Each block is a layer of neurons, and these layers are connected in a sequence, where the output of one layer becomes the input of the next.\nImagine It Like a Sandwich! Let’s imagine a Sequential model is like a sandwich you’re making:\nBread Layer (Input Layer): This is the starting point. You put your base layer. Cheese Layer (Hidden Layer): This is where the \u0026ldquo;magic\u0026rdquo; happens. You add cheese (or whatever filling you like). This layer is where the data is processed and patterns are learned. Top Bread Layer (Output Layer): This is the final result, like putting the top bread on your sandwich. In the Sequential model, you add layers one by one, just like assembling a sandwich.\nExample of a Sequential Model\nfrom tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense # Creating a Sequential model model = Sequential() model.add(Dense(64, activation=\u0026#39;relu\u0026#39;, input_shape=(8,))) # First hidden layer with 64 neurons model.add(Dense(32, activation=\u0026#39;relu\u0026#39;)) # Second hidden layer with 32 neurons model.add(Dense(1)) # Output layer with 1 neuron Explanation in Simple Terms: # model = Sequential(): We create an empty Sequential model.\nImagine starting with an empty plate for your sandwich.\nmodel.add(Dense(64, activation='relu', input_shape=(8,))):\nDense(64) means we’re adding a layer of 64 neurons. activation=\u0026lsquo;relu\u0026rsquo;: We’re using ReLU (Rectified Linear Unit) as the activation function, which helps the neurons decide whether to pass a signal forward or not. Think of ReLU as a light switch — if the input is positive, it passes it on; if negative, it turns it off. input_shape=(8,): This means we expect 8 features as input. For example, if we have 8 features like number of rooms, area, population, etc., this specifies the shape. This is like adding the cheese layer in the sandwich that takes inputs from the bread below.\nmodel.add(Dense(32, activation='relu')):\nAnother layer of 32 neurons is added, again using ReLU activation. This layer helps learn more abstract features from the previous layer. Imagine adding another filling to your sandwich!\nmodel.add(Dense(1):\nOutput layer with 1 neuron because we want a single value as output (house price). This is like putting the final top bread on the sandwich, finishing it off.\nHow the Sequential Model Works: # Input Layer: Takes in the raw features (like number of rooms, area, etc.). Hidden Layers: Each hidden layer transforms the input data, learning different aspects of the data. Neurons in these layers act as \u0026ldquo;mini-experts,\u0026rdquo; learning specific patterns. The ReLU activation function helps the model ignore negative values and focus on positive signals. Output Layer: Produces the final value — in our case, the predicted house price. The beauty of the Sequential model is its simplicity — it\u0026rsquo;s a straightforward stack of layers. You add them in order, and each layer does its part to help make the final prediction.\n","date":"23 September 2024","externalUrl":null,"permalink":"/post/ml-faq/","section":"Post","summary":"This page contains an easy explaination of common jargons of machine learning.","title":"Machine Learning FAQ","type":"post"},{"content":" The Problem # On Day 13 of the 30 Day 30 Machine Learning Projects Challenge, the goal was to build a model capable of classifying music into different genres based on audio features. The idea is to process audio files and extract specific characteristics from them, which can then be used to train a machine learning model that can predict genres.\nIf you want to see the code, you can find it here: GIT REPO.\nUnderstanding the Data # We used the GTZAN Music Genre Dataset, which contains 1000 audio files (30 seconds each) across 10 genres: blues, classical, country, disco, hip-hop, jazz, metal, pop, reggae, and rock.\nThe audio files are stored in .wav format, and to train our model, we need to extract key audio features from the files like:\nMFCCs (Mel-frequency cepstral coefficients): Describes the short-term power spectrum of sound, which is crucial for identifying different genres. Spectral Contrast: Measures the difference between peaks and valleys in the sound spectrum. Chroma: Represents the pitch classes (e.g., C, C#, D, etc.) and is useful in recognizing harmonic and melodic elements. Code Workflow # Here’s the step-by-step breakdown of the approach we followed:\nLoad the Data and Extract Features Split the Data into Training and Validation Sets Build and Train the Model Make Predictions and Evaluate Visualize the Results Before we start, import the required libraries:\nimport os import librosa import numpy as np import pandas as pd from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import accuracy_score, confusion_matrix, classification_report import matplotlib.pyplot as plt import seaborn as sns Step 1: Load the Data and Extract Features # We first downloaded the GTZAN Music Genre Dataset from the provided link. The audio files were categorized into subfolders based on their genre. We need only genes_original for this project. Unzip and place it in the dataset folder of your project. I have renamned it to genes. You can keep it if you desire but do change the name in the code:\nCorrupted File Issue: During the process, we encountered an issue with the file /jazz/jazz.00054.wav, which was corrupted and had to be removed from the dataset before training.\ndataset_path = \u0026#39;dataset/genres/\u0026#39; Feature extraction: To convert audio files into data that a machine learning model can understand, we extracted key features using the librosa library. Here\u0026rsquo;s a snippet showing how we extracted MFCCs and other features:\n# Initialize lists to store features and labels. features = [] labels = [] # Loop through each genre folder and load audio files. for genre in os.listdir(dataset_path): genre_path = os.path.join(dataset_path, genre) for file in os.listdir(genre_path): file_path = os.path.join(genre_path, file) # Load the audio file and extract features y, sr = librosa.load(file_path, duration=30) # Load a 30 sec audio clip # Sample output: # y: [ 0.03451538 0.04815674 0.06430054 ... -0.03909302 -0.02001953 0.05392456] # sr: 22050 # Extract features. mfcc = librosa.feature.mfcc(y = y, sr = sr, n_mfcc=13).mean(axis=1) chroma = librosa.feature.chroma_stft(y = y, sr = sr).mean(axis=1) contrast = librosa.feature.spectral_contrast(y = y, sr = sr).mean(axis=1) zcr = librosa.feature.zero_crossing_rate(y).mean() # Append to features labels list features.append(np.hstack([mfcc, chroma, contrast, zcr])) labels.append(genre) Convert the features and labels lists into pandas Dataframe.\nX = pd.DataFrame(features) y = pd.Series(labels) Step 2: Split the Data # We split the data into an 80% training set and a 20% test set, making sure that the training and validation datasets have a balanced distribution of genres.\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) Step 3: Build and Train the Model # We trained a Random Forest Classifier with 100 decision trees. The Random Forest algorithm builds multiple decision trees and averages their predictions to make more accurate classifications.\nmodel = RandomForestClassifier(n_estimators=100, random_state=42) model.fit(X_train, y_train) Step 4: Make Predictions and Evaluate # Once the model was trained, we predicted genres on the validation set and evaluated the results using accuracy, a confusion matrix, and a classification report.\npredictions = model.predict(X_val) accuracy_score = accuracy_score(y_val, predictions) print(\u0026#34;Accuracy Score:\\n\u0026#34;, accuracy_score) confusion_matrix = confusion_matrix(y_val, predictions) print(\u0026#34;Confusion Matrix:\\n\u0026#34;, confusion_matrix) classification_report = classification_report(y_val, predictions, zero_division=1) print(\u0026#34;Classification report:\\n\u0026#34;, classification_report) Step 5: Visualization # plt.figure(figsize=(10, 7)) sns.heatmap(confusion_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=y.unique(), yticklabels=y.unique()) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Confusion Matrix for Music Genre Classification\u0026#39;) plt.show() Model Performance # Accuracy Score: 0.635 Confusion Matrix: [[17 0 1 1 0 1 0 0 1 1] [ 0 26 1 0 1 0 0 0 0 0] [ 2 0 13 2 0 0 0 1 3 1] [ 2 0 1 12 3 1 0 0 0 5] [ 0 0 0 2 11 0 1 1 2 3] [ 2 3 0 1 0 10 0 0 3 0] [ 1 0 0 0 0 0 11 0 0 0] [ 0 0 2 1 1 2 0 14 1 0] [ 1 0 2 0 4 0 0 1 7 0] [ 4 0 2 5 0 0 0 0 0 6]] Classification report: precision recall f1-score support blues 0.59 0.77 0.67 22 classical 0.90 0.93 0.91 28 country 0.59 0.59 0.59 22 disco 0.50 0.50 0.50 24 hiphop 0.55 0.55 0.55 20 jazz 0.71 0.53 0.61 19 metal 0.92 0.92 0.92 12 pop 0.82 0.67 0.74 21 reggae 0.41 0.47 0.44 15 rock 0.38 0.35 0.36 17 accuracy 0.64 200 macro avg 0.64 0.63 0.63 200 weighted avg 0.64 0.64 0.63 200 With Accuracy: 63.5%, you can see that the model performed best on classical and metal genres but struggled with rock and disco.\nGratitude # Today\u0026rsquo;s project helped me understand how audio features can be used to classify music genres. This was the second problem I solved using the RandomForestClassifier model, and remembering key points from the last problem made me feel like learning while coding is really working. I’m excited to solve more problems.\nStay Tuned!\n","date":"23 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-13/","section":"Challenges","summary":"On Day 13 of the 30 Day 30 Machine Learning Projects Challenge, the task was to build a music genre classifier using audio features. We aimed to classify songs into different genres based on their audio characteristics. We used machine learning to recognize patterns and differentiate between genres like rock, pop, jazz, and more.","title":"Day 13 - 30 Days 30 ML Projects: Build a Music Genre Classifier using Audio Features Extraction","type":"challenge"},{"content":" Problem: Predicting Airline Passenger Satisfaction with Gradient Boosting Machine (GBM) # Hey, it’s Day 12 of the 30 Day 30 Machine Learning Projects Challenge! Today, the task was to predict whether airline passengers were satisfied or dissatisfied with their flight experience using a Gradient Boosting Machine (GBM) model. Let\u0026rsquo;s walk through how we solved this problem, what GBM is, and why we used One-Hot Encoding to prepare our data.\nIf you want to see the code, you can find it here: GIT REPO.\nThe Problem # The challenge was to predict airline passenger satisfaction using features such as flight distance, in-flight services, seat class, and more. The goal was to build a machine learning model to classify passengers as satisfied or dissatisfied based on their flight experience.\nUnderstanding the Data # We used the Airline Passenger Satisfaction Dataset from Kaggle. The dataset includes columns such as Age, Flight Distance, Type of Travel (business or personal), Class (Economy, Business), and in-flight service ratings. The target column, \u0026lsquo;satisfaction\u0026rsquo;, had values:\nSatisfied (1) Neutral or Dissatisfied (0) Download and place it in the dataset directory of your project.\nCode Workflow # The process was divided into several steps:\nLoad the data Preprocess the data using One-Hot Encoding Build and Train the Gradient Boosting Model Make Predictions and Evaluate Visualization Step 1: Load the Data # We loaded the training and test datasets from the dataset directory:\ntrain_data = pd.read_csv(\u0026#39;dataset/airline_passenger_satisfaction_train.csv\u0026#39;) test_data = pd.read_csv(\u0026#39;dataset/airline_passenger_satisfaction_test.csv\u0026#39;) Step 2: Preprocess the Data # We applied One-Hot Encoding to convert the categorical variables (like Gender, Class, Type of Travel) into binary columns that machine learning models can understand.\nWhat is One-Hot Encoding?\nOne-Hot Encoding is a method used to convert categorical variables into binary (0/1) columns. For example, if a column called \u0026lsquo;Class\u0026rsquo; has values like \u0026ldquo;Business\u0026rdquo;, \u0026ldquo;Eco\u0026rdquo;, and \u0026ldquo;Eco Plus\u0026rdquo;, one-hot encoding will create new columns such as \u0026lsquo;Class_Business\u0026rsquo;, \u0026lsquo;Class_Eco\u0026rsquo;, and \u0026lsquo;Class_Eco Plus\u0026rsquo;. Each row will have a value of 1 for the column corresponding to the category that applies.\nExample: Class Eco will be transformed to:\nClass_Business: 0 Class_Eco: 1 Class_Eco Plus: 0 X_train = pd.get_dummies(train_data.drop(\u0026#39;satisfaction\u0026#39;, axis=1), drop_first=True) y_train = train_data[\u0026#39;satisfaction\u0026#39;].map({\u0026#39;satisfied\u0026#39;: 1, \u0026#39;neutral or dissatisfied\u0026#39;: 0}) X_val = pd.get_dummies(test_data.drop(\u0026#39;satisfaction\u0026#39;, axis=1), drop_first=True) y_val = test_data[\u0026#39;satisfaction\u0026#39;].map({\u0026#39;satisfied\u0026#39;: 1, \u0026#39;neutral or dissatisfied\u0026#39;: 0}) Step 3: Build and Train the Model # We used a Gradient Boosting Machine (GBM), which is a powerful algorithm that builds multiple models (weak learners) sequentially. Each new model tries to correct the errors made by the previous ones, resulting in improved accuracy.\nWhat is Gradient Boosting Machine (GBM)?\nGBM is an ensemble learning method where decision trees are built sequentially. Each tree corrects the errors of the previous trees. The idea is to boost weak models (small trees) into a strong predictive model. It works well for both classification and regression tasks.\nWe set random_state=42 to ensure that the results are consistent every time we run the code.\nmodel = GradientBoostingClassifier(random_state=42) model.fit(X_train, y_train) Step 4: Make Predictions and Evaluate # We made predictions on the validation set and evaluated the model using accuracy score, confusion matrix, and classification report.\npredictions = model.predict(X_val) accuracy_score = accuracy_score(y_val, predictions) print(\u0026#34;Accuracy Score:\\n\u0026#34;, accuracy_score) confusion_matrix = confusion_matrix(y_val, predictions) print(\u0026#34;Confusion Matrix:\\n\u0026#34;, confusion_matrix) classification_report = classification_report(y_val, predictions) print(\u0026#34;Classification Report:\\n\u0026#34;, classification_report) Here\u0026rsquo;s the output:\nAccuracy Score: 0.9418375622755185 Confusion Matrix: [[13919 609] [ 897 10468]] Classfication Report: precision recall f1-score support 0 0.94 0.96 0.95 14528 1 0.95 0.92 0.93 11365 accuracy 0.94 25893 macro avg 0.94 0.94 0.94 25893 weighted avg 0.94 0.94 0.94 25893 Step 5: Visualization # Confusion Matrix Heatmap\nWe used a heatmap to visualize the confusion matrix, showing how well the model classified satisfied vs dissatisfied passengers.\nplt.figure(figsize=(7,5)) sns.heatmap(confusion_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=[\u0026#39;Dissatisfied\u0026#39;, \u0026#39;Satisfied\u0026#39;], yticklabels=[\u0026#39;True Dissatisfied\u0026#39;, \u0026#39;True Satisfied\u0026#39;]) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Confusion Matrix\u0026#39;) plt.show() Feature Importance Visualization\nFeature importance helps us understand which features (e.g., age, flight distance) have the most impact on the model\u0026rsquo;s predictions.\nfeature_importance = model.feature_importances_ sorted_idx = np.argsort(feature_importance) plt.figure(figsize=(15,8)) plt.barh(X_train.columns[sorted_idx], feature_importance[sorted_idx], color=\u0026#39;teal\u0026#39;) plt.xlabel(\u0026#39;Importance\u0026#39;) plt.ylabel(\u0026#39;Features\u0026#39;) plt.title(\u0026#39;Feature Importance for Passenger Satisfaction Prediction\u0026#39;) plt.show() Model Performance # With an accuracy of 94%, the Gradient Boosting Machine (GBM) model performed well in predicting airline passenger satisfaction. It correctly identified most of the satisfied and dissatisfied passengers, with some missed cases. From the confusion matrix heatmap we can see:\n13,919 True Dissatisfied passengers were correctly identified. 10,468 True Satisfied passengers were correctly identified. 609 Dissatisfied passengers were misclassified as satisfied. 897 Satisfied passengers were misclassified as dissatisfied. Gratitude # It was exciting to work with Gradient Boosting and see the role of each feature in predicting passenger satisfaction. Looking forward to solving more problems in this challenge!\nStay tuned!\n","date":"22 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-12/","section":"Challenges","summary":"Hey, It is Day 11 of the 30 Days 30 Machine Learning Project Challenge. The problem is to Predict Airline Passenger Satisfaction with Gradient Boosting Machine (GBM). Curious about how it went? Read on to see the results!","title":"Day 12 - 30 Days 30 Machine Learning Projects Challenge","type":"challenge"},{"content":" The Problem # On Day 11 of the 30 Day 30 Machine Learning Projects Challenge, we focused on detecting credit card fraud using an Isolation Forest model. The goal was to identify anomalies in transaction data, labeling these anomalies as potential fraud cases.\nIf you want to see the code, you can find it here: GIT REPO.\nUnderstanding the Data # We used the Credit Card Fraud Detection Dataset from Kaggle. The dataset includes transactions labeled as either normal (0) or fraud (1). In this project, we used Isolation Forest to separate the normal and fraudulent transactions.\nCode Workflow # The steps involved were as follows:\nLoad the Data Create Feature and Target datasets Split the Data Build and Train the Model Make Predictions and Evaluate Visualization Step 1: Load the data # Download the data from kaggle and put it in the dataset directory at the root of your project.\ndata = pd.read_csv(\u0026#39;dataset/creditcard.csv\u0026#39;) Step 2: Create Feature and Target Datasets # We separated the features (X) and target labels (y). Additionally, we mapped the target labels for consistency with the Isolation Forest model, where 1 represents normal transactions and -1 represents fraud (anomalies).\nX = data.drop(\u0026#39;Class\u0026#39;, axis=1) y = data[\u0026#39;Class\u0026#39;].map({0: 1, 1: -1}) # 1: Normal, -1: Fraud/Anomaly Step 3: Split the Data # We split the dataset into 80% training and 20% validation sets. To ensure the distribution of normal and fraud transactions remains balanced across the training and validation sets, we used stratification:\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y) Step 4: Build and Train the Isolation Forest Model # We used Isolation Forest, which is an unsupervised algorithm for anomaly detection. The contamination parameter was set to 0.01 (assuming 1% of the data are anomalies).\nmodel = IsolationForest(contamination=0.01, random_state=42) model.fit(X_train) Step 5: Make Predictions and Evaluate # The model predicted whether each transaction was normal (1) or an anomaly (-1). We used a confusion matrix and other metrics to evaluate the model\u0026rsquo;s performance.\npredictions = model.predict(X_val) X_val[\u0026#39;anomaly\u0026#39;] = predictions accuracy = accuracy_score(y_val, predictions) conf_matrix = confusion_matrix(y_val, predictions) class_report = classification_report(y_val, predictions, zero_division=1) print(f\u0026#34;Accuracy Score: {accuracy}\u0026#34;) print(f\u0026#34;Confusion Matrix:\\n{conf_matrix}\u0026#34;) print(f\u0026#34;Classification Report:\\n{class_report}\u0026#34;) Key Metrics:\nTrue Positives (fraud correctly identified as fraud) False Positives (normal transactions mistakenly flagged as fraud) False Negatives (fraud missed by the model) True Negatives (normal transactions correctly identified) Step 6: Visualization # We created a scatter plot to visualize the anomalies versus normal transactions based on two features (V1 and V2), and used a confusion matrix heatmap to show the model\u0026rsquo;s performance.\nScatter Plot:\nplt.figure(figsize=(10, 6)) plt.scatter(X_val[\u0026#39;V1\u0026#39;], X_val[\u0026#39;V2\u0026#39;], c=predictions, cmap=\u0026#39;coolwarm\u0026#39;, label=\u0026#39;Anomalies\u0026#39;) plt.xlabel(\u0026#39;V1\u0026#39;) plt.ylabel(\u0026#39;V2\u0026#39;) plt.title(\u0026#39;Isolation Forest: Anomalies vs Normal Transactions\u0026#39;) plt.legend() plt.show() Confusion Matrix Heatmap:\nplt.figure(figsize=(10, 6)) sns.heatmap(conf_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;, xticklabels=[\u0026#39;Anomaly\u0026#39;, \u0026#39;Normal\u0026#39;], yticklabels=[\u0026#39;True Anomaly\u0026#39;, \u0026#39;True Normal\u0026#39;]) plt.xlabel(\u0026#39;Predicted\u0026#39;) plt.ylabel(\u0026#39;Actual\u0026#39;) plt.title(\u0026#39;Confusion Matrix Heatmap\u0026#39;) plt.show() Model Performance # With 99% of accuracy the Isolation Forest model performed well in detecting anomalies. We can see a high rate of true normal transactions but some missed fraud cases and false positives.\nAccuracy Score: 0.989431550858467 Confusion Matrix: [[ 53 45] [ 557 56307]] Classfication Report: precision recall f1-score support -1 0.09 0.54 0.15 98 1 1.00 0.99 0.99 56864 accuracy 0.99 56962 macro avg 0.54 0.77 0.57 56962 weighted avg 1.00 0.99 0.99 56962 53 True Anomalies (Fraud) were correctly identified. 45 Fraud Cases were missed by the model. 557 False Positives were normal transactions mistakenly flagged as fraud. 56,307 True Normals were correctly identified as normal transactions. Gratitude # Working with unsupervised learning and anomaly detection was a great learning experience. Stay tuned for Day 12!\n","date":"21 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-11/","section":"Challenges","summary":"Hey, It is Day 11 of the 30 Days 30 Machine Learning Project Challenge. The problem is to detect anomalies in credit card transaction data with isolation forest. Curious about how it went? Read on to see the results!","title":"Day 11 - 30 Days 30 Machine Learning Projects: Anomaly Detection with Isolation Forest","type":"challenge"},{"content":"Hey, it’s Day 10 of the 30 Day 30 Machine Learning Projects Challenge. Today’s task was to build a Recommender System using Collaborative Filtering on a user-item ratings matrix. This was an exciting challenge that helped me understand how recommendation engines like the ones used by Netflix and Amazon work!\nIf you want to see the code, you can find it here: GIT REPO.\nThe Problem # The goal today was to predict how users would rate movies that they haven’t watched yet, based on the ratings they’ve given to other movies. This was done using Collaborative Filtering, a popular technique in recommendation systems.\nWhat is Collaborative Filtering? # Collaborative Filtering is a method used by recommender systems to suggest items to users by looking at the preferences of similar users or similar items. There are two main types of collaborative filtering:\nUser-Based Collaborative Filtering: Recommends items to a user based on items liked by similar users. Item-Based Collaborative Filtering: Recommends items similar to the ones the user has already liked. For this project, I implemented Item-Based Collaborative Filtering, which focuses on finding similarities between movies based on user ratings and making predictions accordingly.\nCosine Similarity # To determine how similar two movies are, I used Cosine Similarity. This is a metric that measures how similar two vectors (in this case, movie ratings) are by calculating the angle between them.\nIf two movies are rated similarly by users, their cosine similarity will be close to 1 (very similar). If two movies have very different ratings, their cosine similarity will be closer to 0 (not similar). The formula for cosine similarity is: cosine_similarity(A, B) = A⋅B / ∣∣A∣∣×∣∣B∣∣ ​ Where:\nA and B are the rating vectors for two movies. The dot product is the sum of the product of corresponding elements from the two vectors. The denominator normalizes the values to account for the magnitudes of the vectors. Approach and Code Workflow # Step 1: Load the Data # I used the MovieLens dataset from Kaggle, which contains user ratings for movies. This dataset has information on users, movies, and the ratings given by users to different movies. Download, unzip and put it in the dataset directory of your project.\nimport pandas as pd # Load the ratings dataset ratings = pd.read_csv(\u0026#39;dataset/ml-latest-small/ratings.csv\u0026#39;) # Load the movies dataset (optional for movie names) movies = pd.read_csv(\u0026#39;dataset/ml-latest-small/movies.csv\u0026#39;) Step 2: Create the User-Item Matrix # I created a matrix where rows represent users, columns represent movies, and the values represent the ratings given by users to movies.\nuser_item_matrix = ratings.pivot(index=\u0026#39;userId\u0026#39;, columns=\u0026#39;movieId\u0026#39;, values=\u0026#39;rating\u0026#39;) user_item_matrix.fillna(0, inplace=True) Step 3: Calculate Cosine Similarity # To recommend movies based on similar ones, I used Cosine Similarity to calculate how similar the movies are based on their ratings.\nfrom sklearn.metrics.pairwise import cosine_similarity # Calculate the cosine similarity between items (movies) item_similarity = cosine_similarity(user_item_matrix.T) # Transpose to get movie-to-movie similarity item_similarity_df = pd.DataFrame(item_similarity, index=user_item_matrix.columns, columns=user_item_matrix.columns) Step 4: Make Predictions Based on Similarity # To predict how a user would rate a movie they haven’t rated yet, I used the similarity between movies and the ratings the user has given to similar movies.\nimport numpy as np # Predict ratings def predict_ratings(user_item_matrix, similarity_matrix): return np.dot(user_item_matrix, similarity_matrix) / np.abs(similarity_matrix).sum(axis=1) # Make predictions using item similarity predicted_ratings = predict_ratings(user_item_matrix.values, item_similarity) # Convert the predictions back into a DataFrame for readability predicted_ratings_df = pd.DataFrame(predicted_ratings, index=user_item_matrix.index, columns=user_item_matrix.columns) Step 5: Evaluate the Model # I evaluated the model using Root Mean Squared Error (RMSE). RMSE tells us how far off our predicted ratings are from the actual ratings. The lower the RMSE, the better the model.\nfrom sklearn.metrics import mean_squared_error # Flatten the matrices and calculate RMSE true_ratings = user_item_matrix.values.flatten() predicted_ratings = predicted_ratings_df.values.flatten() # Calculate RMSE rmse = np.sqrt(mean_squared_error(true_ratings[true_ratings \u0026gt; 0], predicted_ratings[true_ratings \u0026gt; 0])) print(f\u0026#34;Root Mean Squared Error: {rmse}\u0026#34;) Unfortunately, the RMSE came out to be 9.89, which is quite high, given that the ratings in the dataset range from 1 to 5. This suggests that the model’s predictions were not very accurate.\nModel Performance # The RMSE value of 9.89 means the predicted ratings are quite far off from the actual ratings, indicating that this simple collaborative filtering model isn’t performing very well. There are several potential improvements we could make, such as:\nUsing Advanced Algorithms: Models like Matrix Factorization (SVD) or ALS (Alternating Least Squares) handle sparse data better and could reduce the error. Feature Engineering: We could add additional features, such as user preferences, genres, or movie popularity, to improve the accuracy of the predictions. Gratitude # This project was a great learning experience, even though the model didn’t perform as expected. I’m looking forward to diving deeper into more advanced recommendation algorithms in future projects.\nStay tuned!\n","date":"20 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-10/","section":"Challenges","summary":"Hey, It is Day 10 of the 30 Days 30 Machine Learning Project Challenge. The problem is to build recommender system using collaborative filtering. Curious about how it went? Read on to see the results!","title":"Day 10 - 30 Days 30 Machine Learning Projects: Recommender System using Collaborative Filtering","type":"challenge"},{"content":" Problem: Forecasting weather with Simple Linear Regression on time series data # Hey, it’s Day 9 of the 30 Day 30 Machine Learning Projects Challenge. Today’s challenge was about forecasting weather using a Simple Linear Regression model. The goal was to predict future temperatures based on historical temperature data. Let’s break it down step-by-step and see how the model performed.\nIf you want to see the code, you can find it here: GIT REPO.\nUnderstanding the Data # We used the Daily Temperature of Major Cities dataset from Kaggle. It contains temperature data from cities around the world, recorded daily. For this project, we filtered the data to focus on India and used Date and AvgTemperature (average temperature) as the primary columns for forecasting. Download, unzip and put it in the dataset directory at the root level of your project.\nThe data spans multiple years, and the challenge was to predict the temperature for future dates based on past data using a time series approach.\nStep-by-Step Code Workflow # The code was broken down into the following steps:\nStep 1: Load the Data # We started by loading the dataset using pandas. Since the data is large, we set low_memory=False to avoid mixed-type warnings during loading.\ndata = pd.read_csv(\u0026#39;dataset/city_temperature.csv\u0026#39;, low_memory=False) Step 2: Filter Data for India # We filtered the dataset for India and removed any invalid temperature values (AvgTemperature \u0026gt; -99).\nindia_data = data[(data[\u0026#39;Country\u0026#39;] == \u0026#39;India\u0026#39;) \u0026amp; (data[\u0026#39;AvgTemperature\u0026#39;] \u0026gt; -99)].copy() Step 3: Combine Date Columns # Next, we combined the Year, Month, and Day columns into a single Date column to create a proper time series.\nindia_data.loc[:, \u0026#39;Date\u0026#39;] = pd.to_datetime(india_data[[\u0026#39;Year\u0026#39;, \u0026#39;Month\u0026#39;, \u0026#39;Day\u0026#39;]]) Step 4: Select Relevant Columns # We only kept the relevant columns — Date and AvgTemperature — for our analysis.\nrel_india_data = india_data[[\u0026#39;Date\u0026#39;, \u0026#39;AvgTemperature\u0026#39;]] Step 5: Preprocess the Data # We removed any missing values to ensure clean data, then converted the Date into a numeric format using ordinal numbers. This allowed our Linear Regression model to work with time as a feature.\nrel_india_data = rel_india_data.dropna() rel_india_data[\u0026#39;Date_ordinal\u0026#39;] = rel_india_data[\u0026#39;Date\u0026#39;].map(pd.Timestamp.toordinal) X = rel_india_data[[\u0026#39;Date_ordinal\u0026#39;]] # Feature y = rel_india_data[\u0026#39;AvgTemperature\u0026#39;] # Target Step 6: Train-Test Split # We split the data into training (80%) and validation (20%) datasets using train_test_split.\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) Step 7: Build and Train the Model # We trained a Simple Linear Regression model using the training data. The model tries to fit a straight line to the relationship between time and temperature.\nmodel = LinearRegression() model.fit(X_train, y_train) Step 8: Make Predictions and Evaluate # The model made predictions on the validation data. We evaluated its performance using Mean Squared Error (MSE), which measures how far the predicted values are from the actual values.\npredictions = model.predict(X_val) mean_squared_error = mean_squared_error(y_val, predictions) print(mean_squared_error) Step 9: Visualization # We visualized the model\u0026rsquo;s predictions against the actual temperatures. The blue dots represent actual temperatures, while the red line represents the predicted temperatures.\nplt.figure(figsize=(7, 5)) plt.scatter(X_val, y_val, color=\u0026#39;blue\u0026#39;, label=\u0026#39;Actual Temperature\u0026#39;) plt.plot(X_val, predictions, color=\u0026#39;red\u0026#39;, linewidth=2, label=\u0026#39;Predicted Temperature\u0026#39;) plt.xlabel(\u0026#39;Date (Ordinal)\u0026#39;) plt.ylabel(\u0026#39;Temperature\u0026#39;) plt.title(\u0026#39;Weather Forecast for India: Actual vs Predicted\u0026#39;) plt.legend() plt.show() Model Performance # The model achieved an accuracy of 74%, which is moderate. However, the performance is not ideal, as weather patterns can be very complex and linear models often fail to capture these trends accurately.\nPrediction Line: The predicted temperatures were nearly constant, as seen in the plot. This is a limitation of the linear model, as it struggles to capture non-linear, seasonal trends in weather data. What Can We Do? # To improve the forecast, here are a few options to explore:\nIntroduce Complexity: We might need more sophisticated models like Polynomial Regression (to capture nonlinear trends) or Time Series Models like ARIMA or Prophet, which can account for seasonal patterns. Add More Features: Simple Linear Regression is based only on the date. Adding additional features, such as previous day’s temperature, humidity, or atmospheric pressure, might help the model capture more intricate weather patterns. Key Takeaways # Simple Linear Regression can capture basic trends, but it struggles with complex data like weather forecasting.\nGratitude # Working on weather forecasting with time series data was a great experience. It highlighted the limitations of linear models for complex patterns like weather and gave me insight into how we can tackle such problems using more advanced techniques.\nStay tuned!\n","date":"19 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-9/","section":"Challenges","summary":"Hey, It is Day 9 of the 30 Days 30 Machine Learning Project Challenge. The problem is to Forecast weather with Simple Linear Regression on time series data. Curious about how it went? Read on to see the results!","title":"Day 9 - 30 Days 30 Machine Learning Projects","type":"challenge"},{"content":"Hey, it’s Day 8 of the 30 Day 30 Machine Learning Projects Challenge. Today’s task was to build a model to detect fake news using a PassiveAggressive Classifier and TfidfVectorizer. Let’s go step-by-step to see how we used machine learning to classify news articles as real or fake.\nIf you want to see the code, you can find it here: GIT REPO.\nThe Problem # The challenge today was to automatically detect fake news by analyzing the content of news articles. The model should predict whether a news article is real or fake.\nUnderstanding the Data # We used the Fake News Detection Datasets Dataset from Kaggle, It contains two datasets: True.csv (real news) and Fake.csv (fake news). Each file contains columns like title, text, subject, and date. In this project, we used the text column to help the model decide if the news is real or fake. Unzip and put it in the dataset directory at the root level of your project.\nCode Workflow # Load the data Prepare the data Preprocess the data: Create feature and target datasets Convert the text data into numerical form using TfidfVectorizer Split the data Built and train model Made predictions and evaluate Visualization Step 1: Load the Data # We first loaded the True.csv and Fake.csv datasets using pandas:\nfake_df = pd.read_csv(\u0026#39;dataset/Fake.csv\u0026#39;) true_df = pd.read_csv(\u0026#39;dataset/True.csv\u0026#39;) Step 2: Prepare the data # I labeled the real news as 1 and the fake news as 0. Then, I combined both datasets into one:\nfake_df[\u0026#39;label\u0026#39;] = 0 # Label 0 is for Fake news true_df[\u0026#39;label\u0026#39;] = 1 # Label 1 is for True news df = pd.concat([true_df, fake_df], axis=0).reset_index(drop=True) Step 3: Preprocess the data # I then separated the data into Features (X) and Target (y) datasets.\nX = df[\u0026#39;text\u0026#39;] # Feature y = df[\u0026#39;label\u0026#39;] # Target Step 4: Convert the text data into numerical form using TfidfVectorizer # To convert the text data into a format that a machine learning model can understand, I used TfidfVectorizer. This method gives weight to words based on how important and unique they are in the dataset.\ntf_idf_vectorizer = TfidfVectorizer(stop_words=\u0026#34;english\u0026#34;, max_df=0.7) # Ignore if the words appears in 70% or more of the documents. X_tf_idf = tf_idf_vectorizer.fit_transform(X) Let\u0026rsquo;s understand TfidfVectorizer in depth:\nHow Does TfidfVectorizer Work? # TfidfVectorizer stands for Term Frequency-Inverse Document Frequency Vectorizer. It\u0026rsquo;s a method of converting textual data (like news articles, emails, or any text) into numerical form (a matrix of numbers) that machine learning algorithms, such as the PassiveAggressiveClassifier, can work with.\nIt combines two key ideas:\nTerm Frequency (TF): Measures how often a word appears in a document.\nTF(t,d) = Number of times term t appears in document d / Total number of terms in document d Example: If the word \u0026ldquo;news\u0026rdquo; appears 5 times in a 100-word document, its TF is: TF(\u0026quot;news\u0026quot;,d) = 5 / 100 = 0.05\nInverse Document Frequency (IDF): Measures how important a word is across all documents. Common words like \u0026ldquo;the\u0026rdquo; get lower scores.\nIDF(t)=log( Total number of documents /Number of documents containing the term t​ ) Words that appear in fewer documents get a higher IDF score.\nHow Does TfidfVectorizer Help? # TfidfVectorizer creates a matrix where:\nRows represent documents (news articles). Columns represent words. Values are the TF-IDF scores, which highlight important words like \u0026ldquo;fraud\u0026rdquo; or \u0026ldquo;scandal\u0026rdquo; and reduce the impact of common words like \u0026ldquo;the\u0026rdquo;. How Does TfidfVectorizer Help the PassiveAggressiveClassifier? # The PassiveAggressiveClassifier uses these TF-IDF scores to detect fake news:\nPassive: The model doesn’t change if it predicts correctly. Aggressive: It updates itself when it makes a mistake to improve future predictions. TF-IDF ensures important words get more attention, helping the classifier focus on key features to decide if the news is fake or real.\nStep 5: Split the Data # We divided the data into an 80-20 ratio: training (80%) and validation (20%) sets using:\nX_train, X_val, y_train, y_val = train_test_split(X_tf_idf, y, test_size=0.2, random_state=42) Here, random_state=42 sets the seed for randomness. This ensures the same split occurs on every run. The number 42 is commonly used but has no special meaning.\nStep 6: Build and Train the Model # I used a PassiveAggressiveClassifier for this task. This model is efficient and updates itself quickly when it makes mistakes, which is why it’s great for real-time detection tasks like fake news.\nmodel = PassiveAggressiveClassifier(max_iter=10) model.fit(X_train, y_train) Step 7: Make Predictions and Evaluate # Once the model was trained, I tested it on the validation data and evaluated how well it performed. Here’s how I checked the accuracy, confusion matrix, and classification report:\npredictions = model.predict(X_val) accuracy_score = accuracy_score(y_val, predictions) print(\u0026#34;Accuracy score:\\n\u0026#34;, accuracy_score) confusion_matrix = confusion_matrix(y_val, predictions) print(\u0026#34;Confusion Matrix:\\n\u0026#34;, confusion_matrix) classification_report = classification_report(y_val, predictions) print(\u0026#34;Classification Report:\\n\u0026#34;, classification_report) Step 8: Visualization # Finally, I created a heatmap to visualize the confusion matrix, which helps us see how well the model predicted real and fake news:\nplt.figure(figsize=(7,5)) sns.heatmap(confusion_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Blues\u0026#39;) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Confusion Matrix\u0026#39;) plt.show() Model Performance # The model achieved an accuracy of 93%, which is a strong result for detecting fake news. Here’s how the model performed:\nTrue Positives (TP): Correctly predicted real news are 4302. True Negatives (TN): Correctly predicted fake news are 4623. False Positives (FP): Predicted fake news as real are 27. False Negatives (FN): Predicted real news as fake are 28. Here’s the classification report:\nClassfication Report: precision recall f1-score support 0 0.99 0.99 0.99 4650 1 0.99 0.99 0.99 4330 accuracy 0.99 8980 macro avg 0.99 0.99 0.99 8980 weighted avg 0.99 0.99 0.99 8980 Gratitude # This was another fun and informative project in the challenge! I’m looking forward to tackling more problems and improving my machine learning skills.\nStay tuned!\n","date":"18 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-8/","section":"Challenges","summary":"Hey, It is Day 8 of the 30 Days 30 Machine Learning Project Challenge. The problem is to Detect fake news with a PassiveAggressive Classifier and TfidfVectorizer. Curious about how it went? Read on to see the results!","title":"Day 8 - 30 Days 30 Machine Learning Projects","type":"challenge"},{"content":"Hey, it’s Day 7 of the 30 Day 30 Machine Learning Projects Challenge. Today’s task was to build a model that predicts whether a customer will default on their credit card payment using a Random Forest Classifier. This is the first problem that uses Random Forest, so let\u0026rsquo;s explore how it works.\nIf you want to go straight to the code, I’ve uploaded it to this repository GIT REPO\nThe Problem # Determine Credit Card defaults using a Random Forest Classifier # We are trying to predict if a customer will default on their credit card payment based on several features, such as their payment history, bill amount, and limit balance. The dataset provides various customer attributes, and the target is whether they defaulted or not.\nWhat is a Random Forest Classifier? # A Random Forest Classifier is a powerful ensemble learning method that builds multiple decision trees and combines their results. Each tree is trained on a random subset of data, and the final prediction is made by averaging the predictions of all trees (in classification tasks, it’s often the majority vote).\nImagine you\u0026rsquo;re asking several friends for their opinion about something. The more friends you ask, the more confident you\u0026rsquo;ll feel about the decision. That\u0026rsquo;s how Random Forest works—each \u0026ldquo;friend\u0026rdquo; (or tree) gives their opinion, and you trust the majority.\nUnderstanding the Data # We used the Credit Card Default Dataset from Kaggle, which contains details about customers\u0026rsquo; financial behavior. The target variable is whether or not they defaulted on their credit card payment. Unzip and put it in the dataset directory at the root level of your project.\nCode Workflow # The process was divided into several steps:\nLoad the data Preprocess the data Split the Data Create and train the Random Forest model Make predictions and evaluate Visualization Step 1: Load the Data # I loaded the credit card dataset using pandas:\ndata_df = pd.read_csv(\u0026#39;dataset/UCI_Credit_Card.csv\u0026#39;) Step 2: Preprocess the data # We separated the features (customer financial attributes) from the target (whether they defaulted or not):\nX = data_df.drop(\u0026#39;default.payment.next.month\u0026#39;, axis=1) # Features y = data_df[\u0026#39;default.payment.next.month\u0026#39;] # target Step 3: Split the Data # We divided the data into an 80-20 ratio: training (80%) and validation (20%) sets using:\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42) Here, random_state=42 sets the seed for randomness. This ensures the same split occurs on every run. The number 42 is commonly used but has no special meaning.\nStep 4: Create and Train the model # I used a Random Forest Classifier with 100 trees (n_estimators=100) to train the model:\nmodel = RandomForestClassifier(n_estimators=200, random_state=42) model.fit(X_train, y_train) The Random Forest learns patterns in the data by building multiple decision trees. Each tree learns from different parts of the data, and the final prediction is made by combining the results of all trees.\nStep 5: Make Prediction and Evaluate # After training, I used the model to predict whether customers in the test set will default. I evaluated the model using the accuracy score, confusion matrix, and classification report.\npredictions = model.predict(X_val) accuracy_score = accuracy_score(y_val, predictions) print(\u0026#34;Accuracy score:\\n\u0026#34;, accuracy_score) confusion_matrix = confusion_matrix(y_val, predictions) print(\u0026#34;Confusion Matrix:\\n\u0026#34;, confusion_matrix) classification_report = classification_report(y_val, predictions) print(\u0026#34;Classfication Report:\\n\u0026#34;, classification_report) Step 6: Visualization # Finally, I visualized the confusion matrix to see how well the model performed on each class:\nplt.figure(figsize=(10, 7)) sns.heatmap(confusion_matrix, annot=True, fmt=\u0026#39;d\u0026#39;) plt.xlabel(\u0026#39;Predicted Values\u0026#39;) plt.ylabel(\u0026#39;Actual Values\u0026#39;) plt.title(\u0026#39;Confusion Matrix\u0026#39;) plt.show() Model Performance # The model achieved an accuracy of 81%, which is quite good for this dataset. However, let’s break it down further:\nTrue Positives (TP): Correctly predicted defaults are 480. True Negatives (TN): Correctly predicted non-defaults 4417. False Positives (FP): Customers predicted to default but didn’t are 270. False Negatives (FN): Customers predicted not to default but actually did 833. Here’s the classification report that provides detailed precision, recall, and F1-scores:\nprecision recall f1-score support 0 0.84 0.94 0.89 4687 1 0.64 0.37 0.47 1313 accuracy 0.82 6000 macro avg 0.74 0.65 0.68 6000 weighted avg 0.80 0.82 0.80 6000 Improvement # We can potentially improve the accuracy by tunning the model:\nHyperparameter Tuning: Random Forests have several parameters, like the number of trees (n_estimators), maximum depth (max_depth), minimum samples per leaf (min_samples_leaf), etc. Tuning these can improve performance.\nmodel = RandomForestClassifier(n_estimators=400, max_depth=20, min_samples_leaf=8, random_state=42) Feature Importance: Check which features contribute the most using feature_importances_. You can try feature selection to remove irrelevant features and possibly improve the model.\nHandling Class Imbalance: If your dataset is imbalanced (imbalance occurs when one class is significantly more frequent than the other, which can bias the model toward the majority class.\u0026quot;), you can adjust class weights using class_weight=\u0026lsquo;balanced\u0026rsquo; in the RandomForestClassifier to give more weight to the minority class.\nKey Takeaways # Random Forest is a robust algorithm that performs well in classification tasks like predicting credit card defaults. Scaling isn’t required for Random Forest, but hyperparameter tuning and handling class imbalance can improve the results. The model shows good accuracy but struggles a bit with classifying the minority class (defaults). In future projects, I will experiment with class balancing techniques to improve this. Gratitude # Today was a great learning experience with Random Forests. I can\u0026rsquo;t wait to finish this challenge and see everything I\u0026rsquo;ve learned.\nStay Tuned!\n","date":"17 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-7/","section":"Challenges","summary":"Hey, It is Day 7 of the 30 Days 30 Machine Learning Project Challenge. The problem is to Determine Credit Card defaults using a Random Forest Classifier. Curious about how it went? Read on to see the results!","title":"Day 7 - 30 Days 30 Machine Learning Projects","type":"challenge"},{"content":"Hey, it’s Day 6 of the 30 Day 30 Machine Learning Projects Challenge. Today’s problem was “Predict wine quality from physicochemical properties using SVM”. This is the 5th classification problem in a row. We will learn what SVM is and how it works, along with other important machine learning techniques.\nIf you want to go straight to the code, I’ve uploaded it to this repository GIT REPO\nThe process will be the same as I briefly explained in the previous progress posts. I’ll use ChatGPT and ask follow-up questions.\nTalk about the Problem Please! # Today, we used Support Vector Machines (SVM) to predict the quality of wine based on its physicochemical properties (like acidity, sugar, and alcohol content). The goal was to build a model that could classify wine into different quality categories (from 0 to 10) using SVM.\nWhat is SVM? # Support Vector Machines (SVM) are powerful classifiers that find the best boundary (hyperplane) between different classes. Imagine you have data points scattered in space, and you need to separate them into different groups. SVM draws a line (in 2D) or a plane (in 3D) that best divides these points.\nIn our case, we used SVC (Support Vector Classifier), a type of SVM designed for classification tasks. To handle the non-linearity of our data, we used the RBF (Radial Basis Function) kernel, which creates curved decision boundaries to separate complex data.\nUnderstanding the Data # We used the Wine Quality Dataset from Kaggle, which contains the physicochemical properties of wine and the corresponding quality ratings. The features include attributes like acidity, sugar levels, and alcohol content, and the target is the wine quality score. Download it locally and put it in the dataset directory at root level of this repository.\nCode Workflow # The process was divided into several steps:\nLoad the data Preprocess the data Data Preprocessing: Feature scaling Split the data into training and validation sets Create and train the SVM model Make predictions and evaluate the model Visualization Step 1: Load the Data # I loaded the wine quality dataset using pandas:\ndata_df = pd.read_csv(\u0026#39;dataset/WineQT.csv\u0026#39;, sep=\u0026#39;,\u0026#39;) Here’s how it looks:\nfixed acidity volatile acidity citric acid residual sugar chlorides free sulfur dioxide total sulfur dioxide density pH sulphates alcohol quality Id 0 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.9978 3.51 0.56 9.4 5 0 1 7.8 0.88 0.00 2.6 0.098 25.0 67.0 0.9968 3.20 0.68 9.8 5 1 2 7.8 0.76 0.04 2.3 0.092 15.0 54.0 0.9970 3.26 0.65 9.8 5 2 3 11.2 0.28 0.56 1.9 0.075 17.0 60.0 0.9980 3.16 0.58 9.8 6 3 4 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.9978 3.51 0.56 9.4 5 4 Step 2: Preprocess the Data # We separated the features (physicochemical properties) from the target (quality):\nX = data_df.drop(\u0026#39;quality\u0026#39;, axis=1) y = data_df[\u0026#39;quality\u0026#39;] Step 3: Data Preprocessing: Scaling the Data # To make sure all features contribute equally, we applied StandardScaler to standardize the data. This is called scaling transformation, which is the process of transforming your data so that all features (variables) are on a similar scale or range. It’s commonly done in machine learning to ensure that no feature dominates the others simply because of its larger numerical range.\nMore technically, StandardScaler ensures that all features contribute equally by transforming the data to have a mean of 0 and a standard deviation of 1.\nLet’s understand scaling with an example:\nSuppose we have a small dataset with two features: height and weight. The values for these features are in different scales. Height has a larger numerical range than weight.\nHeight (cm): 160, 170, 150, 180, 175 Weight (kg): 65, 70, 55, 85, 75 Before Scaling # Let’s calculate the mean and standard deviation for each feature:\nHeight:\nMean: 167 Standard Deviation: 11.18 Weight:\nMean: 70 Standard Deviation: 10 After Applying StandardScaler: # For each value, we use the formula:\nScaled Value = (Original Value - Mean) / Standard Deviation\nFor example,\nFor height 160, scaled value will be (160 - 167) / 11.18 ~ -0.63 For weight 65, scaled value will be (65 - 70) / 10 ~ -0.5 Here’s how the scaled values would look:\nHeights: -0.63, 0.27, -1.52, 1.16, 0.72 Weights: -0.5, 0, -1.5, 1.5, 0.5 Visualizing the Output # Original Data:\nHeight ranges from 150 to 180 cm. Weight ranges from 55 to 85 kg. After Scaling:\nThe transformed height and weight values are now centered around 0, and their standard deviations are 1. This ensures that the data has zero mean and unit variance, meaning all features are on the same scale. Now, let’s code it up:\nstandard_scale = StandardScaler() X_scaled = standard_scale.fit_transform(X) Step 4: Split the Data # We divided the data into an 80-20 ratio: training (80%) and validation (20%) sets using:\nX_train, X_val, y_train, y_val = train_test_split(X_scaled, y, test_size=0.2, random_state=42) Here, random_state=42 sets the seed for randomness. This ensures the same split occurs on every run. The number 42 is commonly used but has no special meaning.\nStep 5: Create and Train the SVM Model # We used the SVM classifier with the RBF kernel (kernel=\u0026lsquo;rbf\u0026rsquo;). This kernel helps the model deal with non-linear data by creating curved decision boundaries.\nmodel = SVC(kernel=\u0026#39;rbf\u0026#39;) model.fit(X_train, y_train) Step 6: Make Predictions and Evaluate # After training, we used the model to predict the wine quality for the validation set. We calculated accuracy and generated a confusion matrix to understand the model’s performance better.\npredictions = model.predict(X_val) accuracy_score = accuracy_score(y_val, predictions) print(\u0026#34;Accuracy Score: \u0026#34;, accuracy_score) confusion_matrix = confusion_matrix(y_val, predictions) print(\u0026#34;Confusion Matrix: \u0026#34;, confusion_matrix) classification_report = classification_report(y_val, predictions, zero_division=0) print(\u0026#34;Classification Report: \u0026#34;, classification_report) We also used the zero_division=0 parameter to avoid warnings when a certain quality label might not be predicted.\nStep 7: Visualization # Finally, we visualized the confusion matrix using seaborn to see how the model performed across different wine quality levels:\nplt.figure(figsize=(8,7)) sns.heatmap(confusion_matrix, annot=True, fmt=\u0026#39;d\u0026#39;, cmap=\u0026#39;Reds\u0026#39;, xticklabels=sorted(y.unique()), yticklabels=sorted(y.unique())) plt.xlabel(\u0026#39;Predicted Quality\u0026#39;) plt.ylabel(\u0026#39;Actual Quality\u0026#39;) plt.title(\u0026#39;Confusion Matrix\u0026#39;) plt.show() Model Performance # Accuracy Score: 0.6593886462882096 Confusion Matrix: [[ 0 3 3 0 0] [ 0 72 24 0 0] [ 0 27 69 3 0] [ 0 1 15 10 0] [ 0 0 1 1 0]] Classfication Report: precision recall f1-score support 4 0.00 0.00 0.00 6 5 0.70 0.75 0.72 96 6 0.62 0.70 0.65 99 7 0.71 0.38 0.50 26 8 0.00 0.00 0.00 2 accuracy 0.66 229 macro avg 0.41 0.37 0.38 229 weighted avg 0.64 0.66 0.64 229 Key Takeaways # SVM is a powerful algorithm for classification, especially with the RBF kernel, which handles non-linear data effectively. Scaling the features is important to ensure all variables contribute equally. More advanced models or tuning the hyperparameters might improve predictions further. Gratitude # It was a great learning experience working with SVM today. Looking forward to next problem.\nStay Tuned!\n","date":"16 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-6/","section":"Challenges","summary":"Hey, It is Day 6 of the 30 Days 30 Machine Learning Project Challenge. The problem is to Predict wine quality from physicochemical properties using SVM. Curious about how it went? Read on to see the results!","title":"Day 6 - 30 Days 30 Machine Learning Projects","type":"challenge"},{"content":"Good Morning, It\u0026rsquo;s Day 5 of the 30 Day 30 Machine Learning Projects Challenge.\nIf you want to go straight to the code, I’ve uploaded it to this repository GIT REPO\nThe process will be the same as I briefly explained in the previous progress posts. I\u0026rsquo;ll use ChatGPT and ask follow-up questions.\nTalk about the Problem Please! # Today\u0026rsquo;s problem was \u0026ldquo;Filter spam from a collection of emails using Naive Bayes\u0026rdquo;. This is the fourth classification problem in a row. Today we will learn about Naive Bais Classifier.\nNaive Bayes is a simple yet powerful algorithm often used for text classification. It uses probabilities to predict which class (spam or not spam) an email belongs to, based on the words in that email.\nHere’s a simple analogy:\nYou have a box of fruits, and you know how often apples and oranges appear in that box. If someone hands you a fruit, you can guess whether it’s an apple or orange based on the features (like color or size). Similarly, Naive Bayes guesses whether an email is spam or not based on the frequency of certain words like \u0026ldquo;win,\u0026rdquo; \u0026ldquo;free,\u0026rdquo; or \u0026ldquo;meeting.\u0026rdquo;\nUndestanding the Data # Since I didn’t have a large dataset initially, I created a small dataset of 20 emails (10 spam and 10 non-spam) to get started. Each email was labeled as either spam or ham (non-spam). Here\u0026rsquo;s a sample of that dataset:\nSpam: \u0026ldquo;Win a $1000 Walmart gift card! Click here to claim now!\u0026rdquo; Non-Spam: \u0026ldquo;Hey, are you coming to the meeting tomorrow?\u0026rdquo; Code Workflow # The workflow is divided into six steps:\nCreate the Data Preprocessing Convert the text data into numerical features Split data in training and validation sets Create and Train Model Make Predictions and Evaluate Visualization Let\u0026rsquo;s understand each step:\nStep 1: Create the Data # I created a small dataset manually, using a dictionary to represent emails and their labels (spam or non-spam). For this small dataset, I used pandas to load the data into a DataFrame:\ndata = { \u0026#39;label\u0026#39;: [\u0026#39;spam\u0026#39;, \u0026#39;ham\u0026#39;, \u0026#39;spam\u0026#39;, \u0026#39;ham\u0026#39;, \u0026#39;ham\u0026#39;, \u0026#39;spam\u0026#39;, \u0026#39;spam\u0026#39;, \u0026#39;ham\u0026#39;, \u0026#39;ham\u0026#39;, \u0026#39;spam\u0026#39;, \u0026#39;ham\u0026#39;, \u0026#39;spam\u0026#39;, \u0026#39;ham\u0026#39;, \u0026#39;spam\u0026#39;, \u0026#39;ham\u0026#39;, \u0026#39;spam\u0026#39;, \u0026#39;spam\u0026#39;, \u0026#39;ham\u0026#39;, \u0026#39;spam\u0026#39;, \u0026#39;ham\u0026#39;], \u0026#39;email\u0026#39;: [ \u0026#39;Win a $1000 Walmart gift card! Click here to claim now!\u0026#39;, \u0026#39;Hey, are you coming to the party tonight?\u0026#39;, \u0026#39;Congratulations! You have won a free vacation to the Bahamas!\u0026#39;, \u0026#39;Can we reschedule our meeting to 3 PM?\u0026#39;, \u0026#39;Your Amazon order has been shipped.\u0026#39;, \u0026#39;You have been selected for a cash prize! Call now to claim.\u0026#39;, \u0026#39;Urgent! Your account has been compromised, please reset your password.\u0026#39;, \u0026#39;Don’t forget about the doctor’s appointment tomorrow.\u0026#39;, \u0026#39;Your package is out for delivery.\u0026#39;, \u0026#39;Get rich quick by investing in this opportunity. Don’t miss out!\u0026#39;, \u0026#39;Can you send me the latest project report?\u0026#39;, \u0026#39;Exclusive offer! Buy one, get one free on all items.\u0026#39;, \u0026#39;Are you free for lunch tomorrow?\u0026#39;, \u0026#39;Claim your free iPhone now by clicking this link!\u0026#39;, \u0026#39;I’ll call you back in 5 minutes.\u0026#39;, \u0026#39;Get a $500 loan approved instantly. No credit check required!\u0026#39;, \u0026#39;Hurry! Limited-time offer, act now to win a $1000 gift card.\u0026#39;, \u0026#39;Let’s catch up over coffee this weekend.\u0026#39;, \u0026#39;You’ve been pre-approved for a personal loan. Apply today!\u0026#39;, \u0026#39;Meeting reminder for Monday at 10 AM.\u0026#39; ] } data_df = pd.DataFrame(data) Step 2: Preprocess the Data # For binary classification, we mapped spam emails to 1 and non-spam (ham) emails to 0. This step allows the model to understand which class each email belongs to:\ndata_df[\u0026#39;label\u0026#39;] = data_df[\u0026#39;label\u0026#39;].map({\u0026#39;ham\u0026#39;: 0, \u0026#39;spam\u0026#39;: 1}) Step 3: Convert the text data into numerical features. # Emails are in text form, so we need to convert them into numerical data. For this, I used the CountVectorizer from sklearn, which creates a matrix of word counts for each email (a \u0026ldquo;bag of words\u0026rdquo; model).\nvectorizer = CountVectorizer(stop_words=\u0026#39;english\u0026#39;) X = vectorizer.fit_transform(data_df[\u0026#39;email\u0026#39;]) # Convert the text into a bag of words matrix. print(\u0026#34;Vocabulary:\\n\u0026#34;, vectorizer.get_feature_names_out()) print(\u0026#34;Count Matrix\\n\u0026#34;, X.toarray()) This transforms each email into a row of numbers, where each number represents how many times a word from the vocabulary appeared in that email.\nStep 4: Split data # I divided the data into an 80-20 ratio: training (80%) and validation (20%) sets using:\nX_train, X_val, y_train, y_val = train_test_split(X, data_df[\u0026#39;label\u0026#39;], test_size=0.2, random_state=42) Here, random_state=42 sets the seed for randomness. This ensures the same split occurs on every run. The number 42 is commonly used but has no special meaning.\nStep 5: Create and Train Model # For this task, I used the Multinomial Naive Bayes model from sklearn. It’s particularly well-suited for text data:\nmodel = MultinomialNB() model.fit(X_train, y_train) The model learned from the training data, looking at how often certain words (like \u0026ldquo;win,\u0026rdquo; \u0026ldquo;free,\u0026rdquo; or \u0026ldquo;meeting\u0026rdquo;) appeared in spam and non-spam emails.\nStep 6: Make Predictions and Evaluate # After training, I used the model to predict whether the emails in the test set were spam or not. To evaluate the model\u0026rsquo;s performance, I calculated its accuracy and created a confusion matrix:\npredictions = model.predict(X_val) accuracy_score = accuracy_score(y_val, predictions) print(\u0026#34;Accuracy Score:\\n\u0026#34;, accuracy_score) confusion_matrix = confusion_matrix(y_val, predictions) print(\u0026#34;\\nConfusion Matrix:\\n\u0026#34;, confusion_matrix) Accuracy alone doesn’t tell the full story. That’s why I also used a confusion matrix to get a better picture of how the model handled spam and non-spam emails.\nHere’s a breakdown of the confusion matrix:\nTrue Positives (TP): Correctly predicted spam emails. True Negatives (TN): Correctly predicted non-spam emails. False Positives (FP): Non-spam emails incorrectly predicted as spam. False Negatives (FN): Spam emails incorrectly predicted as non-spam. Our model\u0026rsquo;s performance:\nAccuracy Score: 1.0 Confusion Matrix: [[2 0] [0 2]] Step 7: Visualization # I used matplotlib.pyp and seaborn to create a heatmap of the confusion matrix.\nWe can see the model predicts:\nTrue Positives (TP): 2 correctly predicted spam emails True Negatives (TN): 2 correctly predicted non-spam emails Gratitude # The Naive Bayes algorithm is simple yet effective for text classification problems like spam filtering. While the model worked well on this small dataset. I will try it on larger dataset in future posts.\nStay Tuned!!\n","date":"15 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-5/","section":"Challenges","summary":"Hey, It is Day 5 of the 30 Days 30 Machine Learning Project Challenge. The problem is to Filter spam from a collection of emails using Naive Bayes. Curious about how it went? Read on to see the results!","title":"Day 5 - 30 Days 30 Machine Learning Projects","type":"challenge"},{"content":"Good Evening! It\u0026rsquo;s Day 4 of the 30 Day 30 Machine Learning Projects Challenge. I went to the Ganesh Festival Heritage walk in the morning, so I only had time to solve the problem late in the evening.\nIf you want to go straight to the code, I’ve uploaded it to this repository GIT REPO\nThe process will be the same as I briefly explained in the Day 1-3 progress posts. I\u0026rsquo;ll use ChatGPT and ask follow-up questions.\nTalk about the Problem Please! # Today\u0026rsquo;s problem was \u0026ldquo;Diagnose breast cancer as malignant or benign using a Decision Tree\u0026rdquo;. This is the third classification problem in a row. Today we\u0026rsquo;ll use a model called a \u0026ldquo;Decision Tree\u0026rdquo;.\nChatGPT Explains: A Decision Tree is like a flowchart that helps you make decisions step by step. It’s a tool used in machine learning to classify things or predict outcomes based on certain conditions.\nImagine This:\nYou want to decide whether a fruit is an apple or an orange. You start asking simple yes or no questions. For example: Is the fruit round? If yes, ask: Is the fruit orange in color? If yes → It\u0026#39;s an orange. If no → It\u0026#39;s an apple. If no → It’s something else, not an apple or orange. Undestanding the Data # For this problem, you can use the Breast Cancer Wisconsin Dataset, which is often used for classification tasks. It\u0026rsquo;s available in Scikit-learn\u0026rsquo;s datasets module, so we don\u0026rsquo;t need to download it separately.\nCode Workflow # The workflow is divided into six steps:\nLoad the dataset Create feature and target set. Split data in training and validation sets Create and Train Model Make Predictions and Evaluate Visualization Let\u0026rsquo;s understand each step:\nStep 1: Load the dataset # Use load_breast_cancer() from sklearn.datasets\ndata = load_breast_cancer() Step 2: Preprocess the Data # We can use the data as is. But I prefer to load it into a Pandas DataFrame. This helps manage datasets, especially for handling and visualizing data before training the model.\nWe can use data.keys() to see the list of keys the data contains. It has these fields:\ndict_keys([\u0026#39;data\u0026#39;, \u0026#39;target\u0026#39;, \u0026#39;frame\u0026#39;, \u0026#39;target_names\u0026#39;, \u0026#39;DESCR\u0026#39;, \u0026#39;feature_names\u0026#39;, \u0026#39;filename\u0026#39;, \u0026#39;data_module\u0026#39;]) First, we create DataFrames using: X = pd.DataFrame(data.data, columns=data.feature_names) And store the target in: y = data.target\nStep 3: Split data. # I divided the data into an 80-20 ratio: training (80%) and validation (20%) sets using: train_test_split(X, y, test_size=0.2, random_state=42)\nHere, random_state=42 sets the seed for randomness. This ensures the same split occurs on every run. The number 42 is commonly used but has no special meaning.\nStep 4: Create and Train Model # Use DecisionTreeClassifier from sklearn.tree. Then train it on X_train and y_train dataset.\nStep 5: Make Predictions and Evaluate # I used a variable named predictions to store the predicted values for the 20% validation data X_val.\nFor diagnosing breast cancer, accuracy alone isn\u0026rsquo;t enough. We must use a Confusion Matrix for evaluation. It gives a more detailed view of the model\u0026rsquo;s performance, showing:\nTrue Positives (TP): Correctly predicted malignant cases. True Negatives (TN): Correctly predicted benign cases. False Positives (FP): Benign cases incorrectly predicted as malignant (also known as Type I error). False Negatives (FN): Malignant cases incorrectly predicted as benign (also known as Type II error). Using a Confusion Matrix is important because:\nAccuracy Alone is Not Enough: Accuracy tells you the percentage of correct predictions, but in cases where one class is more frequent (e.g., more benign tumors than malignant), accuracy might be misleading.\nClass Imbalance: Breast cancer data may have more benign cases than malignant ones, so the model could predict benign for all cases and still get a high accuracy score, but this would be a bad model.\nPerformance Insights: It helps to identify:\nHow well the model is detecting malignant tumors (minimizing false negatives is critical in medical diagnoses). Whether the model is flagging too many benign cases as malignant (false positives). For more information, check out these resources:\nhttps://youtu.be/jr_BcU4QlNE?si=8vZi-XUbVx8s4AHa https://www.ibm.com/topics/confusion-matrix Our model\u0026rsquo;s performance:\nAccuracy is: 0.9473684210526315 Confusion Matrix: [[40 3] [ 3 68]] Classfication Report: precision recall f1-score support 0 0.93 0.93 0.93 43 1 0.96 0.96 0.96 71 accuracy 0.95 114 macro avg 0.94 0.94 0.94 114 weighted avg 0.95 0.95 0.95 114 Step 6: Visualization # I used matplotlib.pyp and seaborn to create a heatmap of the confusion matrix.\nWe can see the model predicts:\nTrue Positives (TP): 40 correctly predicted malignant cases True Negatives (TN): 68 correctly predicted benign cases Let\u0026rsquo;s look at the descision tree:\nGratitude # It was a busy day, and I didn\u0026rsquo;t have time to solve the problem during the day. I finished all the work late at night. I\u0026rsquo;m extremely happy that the challenge streak is STILL ON. I\u0026rsquo;m looking forward to solving the next problem.\nStay Tuned!!\n","date":"14 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-4/","section":"Challenges","summary":"Hey, It is Day 4 of the 30 Days 30 Machine Learning Project Challenge. The problem is to Diagnose breast cancer as malignant or benign using a Decision Tree. Curious about how it went? Read on to see the results!","title":"Day 4 - 30 Days 30 Machine Learning Projects","type":"challenge"},{"content":"Good Morning! It is Day 3 of the 30 Day 30 Machine Learning Projects Challenge, and it is going great. I woke up at 5:17. All credit goes to my cat, Green, for scratching my head with his paws for his morning hunt. :)\nIf you want to go straight to the code, I’ve uploaded it to this repository GIT REPO\nThe flow is going to be the same as I had briefly explained in the Day 1 and Day 2 progress posts. I will be using ChatGPT and moving forward with follow-up questions.\nTalk about the Problem Please! # The problem of the day was \u0026ldquo;Recognizing handwritten digits with k-Nearest Neighbors on MNIST\u0026rdquo;. It is another classic machine learning problem. Here, we have to predict handwritten digits using the K-Nearest Neighbors Algorithm. It also requires using MNIST data.\nWikipedia: The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems.\nUndestanding the Data # I used the data from scikit-learn\u0026rsquo;s datasets fetch_openml with arguments:\nmnist_784: This indicates that we want the MNIST data in which 28x28 size images are flattened into 784-feature vectors. version=1: I specified that I wanted version 1 of the MNIST data. as_frame=True: I specified that I wanted it in Panda DataFrame format, as it is easier to debug, visualize, and manipulate Panda DataFrames. parser='auto': On my local machine, I was getting a warning about the parser version, so I set it to auto to pick the one that works best for the environment. Code Workflow # The workflow is divided into seven steps:\nLoad the MNIST data Preprocess the Data Normalize the Data Split data in training and validation sets Create and Train Model Make Predictions and Evaluate Visualization Let\u0026rsquo;s understand each step:\nStep 1: Load the MNIST data # I have already mentioned in brief that I am using fetch_openml. See the Understand the Data section.\nStep 2: Preprocess the Data # I had mentioned in Step 1 to load the data as Panda DataFrame, I use mnist_data.keys() to know about the list of keys the data contains. It has the following fields:\ndict_keys([\u0026#39;data\u0026#39;, \u0026#39;target\u0026#39;, \u0026#39;frame\u0026#39;, \u0026#39;categories\u0026#39;, \u0026#39;feature_names\u0026#39;, \u0026#39;target_names\u0026#39;, \u0026#39;DESCR\u0026#39;, \u0026#39;details\u0026#39;, \u0026#39;url\u0026#39;]) I used the data and target to build my features (X) and target (y) sets.\nStep 3: Normalize the Data # When dealing with image data, pixel values can range from 0 to 255 for 8-bit grayscale images. Normalizing these pixel values to the range between 0 and 1 is a common preprocessing step in machine learning tasks, particularly for algorithms that are sensitive to the scale of the input data, like k-Nearest Neighbors (k-NN).\nI did X /= 255.0\nStep 4: Split data. # I divided the data into an 80-20 ratio, that is, traning (80%) and validation (20%) sets using train_test_split(X, y, test_size=0.2, random_state=42)\nHere random_state=42 is used to set the seed for randomness. It will ensure that the same split occurs on every run. The number 42 is a commonly used arbitrary number.\nNo logic behind it.\nStep 5: Create and Train Model # I am using KNeighborsClassifier from scikit-learn neighbors package.\nk-NN is a simple, instance-based learning algorithm that classifies new cases based on the majority votes of the k nearest neighbor samples from the training dataset. The \u0026rsquo;nearest neighbors\u0026rsquo; are determined by a distance metric, typically Euclidean distance. Here, K is user-defined.\nInitially, I chose K=3.\nStep 6: Make Predictions and Evaluate # I used a variable named predictions to store the predicted values against the 20% validation data X_val.\nSince it is a classification type of model, relying on accuracy alone is not sufficient. I used a Confusion Matrix to learn more about the efficiency of the model.\nA Confusion Matrix is helpful because it shows True Positives, False Positives, True Negatives, and False Negatives. I know it can be a little difficult to understand; please use the resources mentioned below to grasp it better.\nhttps://youtu.be/jr_BcU4QlNE?si=8vZi-XUbVx8s4AHa https://www.ibm.com/topics/confusion-matrix Visualization # I used matplotlib.pyp and seaborn to create a heatmap of the confusion matrix. See how it looks.\nOutcome of Experimenting with Different K. # At K=3, Accuracy: 0.9712857142857143 At K=2, Accuracy: 0.9642142857142857 At K=1, Accuracy: 0.972 At K=5, Accuracy: 0.9700714285714286 At K=10, Accuracy: 0.9657857142857142 I decided to stick with K=3.\nGratitude # Today, I was feeling confident writing the code and using libraries. It is the second problem on classification; maybe that has helped. I solved it in under 40 minutes, but then I started experimenting with different K values. It was fun. I am now enjoying the process and looking forward to solving more problems.\nStay Tuned!!\n","date":"13 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-3/","section":"Challenges","summary":"Hey, It is Day 3 of the 30 Days 30 Machine Learning Project Challenge. The problem is to Recognize handwritten digits with k-Nearest Neighbors on MNIST. Curious about how it went? Read on to see the results!","title":"Day 3 - 30 Days 30 Machine Learning Projects","type":"challenge"},{"content":"Hey, it is day 2 day of the 30 days 30 ML projects challenge. I got up at 5:30 am, which was again 30 minutes late than the target.\nIf you want to go straight to the code, I’ve uploaded it to this repository GIT REPO\nFlow # The process for solving the problem is going to be the same. I ask ChatGPT for a solution and decode each line by asking follow-up questions. You can find the video of me coding it live at the bottom.\nStraight to Problem, Please! # The challenge for day two was to \u0026ldquo;Classify Iris flowers into species using Logistic Regression\u0026rdquo;. It\u0026rsquo;s a classification problem.\nRequired Packages:\npip install pandas scikit-learn matplotlib numpy seaborn Understand the Data # I am using the Iris dataset from the scikit-learn package. It consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica, and Iris versicolor), with four features describing the lengths and the widths of the sepals and petals.\nsepal length (cm) sepal width (cm) petal length (cm) petal width (cm) species 0 5.1 3.5 1.4 0.2 setosa 1 4.9 3.0 1.4 0.2 setosa 2 4.7 3.2 1.3 0.2 setosa 3 4.6 3.1 1.5 0.2 setosa 4 5.0 3.6 1.4 0.2 setosa Workflow of the Code # The workflow is divided into seven steps:\nLoading the dataset Preparing the data Splitting the dataset Training the model Making predictions Evaluating the model Visualizing the evaluation results Let\u0026rsquo;s dive into each step:\nStep 1: Load the Dataset # Using the Iris dataset from the scikit-learn package, then converting it into a Pandas DataFrame for ease of manipulation.\niris = load_iris() iris_df = pd.DataFrame(iris.data, columns=iris.feature_names) iris_df[\u0026#39;species\u0026#39;] = pd.Categorical.from_codes(iris.target, iris.target_names) Step 2: Select Features and Target # The DataFrame consists of sepal and petal measurements. We\u0026rsquo;ll use these as features (X) to predict the species or the class (y).\nStep 3: Split the Dataset # I split the data into a training set (80%) and a validation set (20%).\nStep 4: Create and Train the Model # I initialized a LogisticRegression model and set max_iter to 200 so that the training process would converge properly. The higher the max_iter count, the better the chances of minimizing the loss function.\nStep 5: Making Predicton # Now, make predictions on the validation data (X_val).\nStep 6: Evaluating the model # For classification problems, relying solely on accuracy may not be sufficient. Therefore, we use a Confusion Matrix to better judge the performance. This will help us cover True Positive, False Positive, True Negative and False Negative of the prediction.\nHere is the output:\nAccuracy is: 1.0 Confusion Matrix is: [[10 0 0] [ 0 9 0] [ 0 0 11]] Classification report is: precision recall f1-score support setosa 1.00 1.00 1.00 10 versicolor 1.00 1.00 1.00 9 virginica 1.00 1.00 1.00 11 accuracy 1.00 30 macro avg 1.00 1.00 1.00 30 weighted avg 1.00 1.00 1.00 30 Step 6: Visualization # I used seaborn to create the heatmap of the Confusion Matrix, and this is how it looks:\nGratitude # I finished in 1 hour, which was faster than I planned. However, there were many new topics, like the Confusion Matrix, which I did not understand well. I read more about it in detail later in the day.\nBelow are the good source you can use:\nhttps://youtu.be/jr_BcU4QlNE?si=8vZi-XUbVx8s4AHa https://www.ibm.com/topics/confusion-matrix Stay Tuned!!\nVideo # ","date":"12 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-2/","section":"Challenges","summary":"Today marks Day 2 of my 30 Days, 30 Machine Learning Projects Challenge. The problem is to Classify Iris flowers into species using Logistic Regression. Curious about how it went? Read on to see the results!","title":"Day 2 - 30 Days 30 Machine Learning Projects","type":"challenge"},{"content":"Today is the first day of the 30 days 30 ML projects challenge. I got up at 5:30 am, which was 30 minutes later than I planned.\nI recorded my screen to keep track of what I did, which helped me write this post. Now I\u0026rsquo;m thinking about posting it on YouTube like a series of development logs. I’ll share the video link at the end so you can see how I start from scratch. I think that\u0026rsquo;s pretty cool.\nIf you want to go straight to the code, I’ve uploaded it to this repository GIT REPO\nFlow # I planned to read blogs and tutorials for reference. Then, I realized that I could use ChatGPT.\nI asked ChatGPT to help solve the problem, telling it to assume I have a basic understanding of Machine Learning and to start with simple models, getting more complex as we go.\nI\u0026rsquo;m going to use the same context window for future problems. This way, I can make the most of ChatGPT without having to train from scratch each time.\nI typed out each line of the code myself, actually copying it, and made changes where needed. If I didn\u0026rsquo;t understand something, I asked ChatGPT to clarify. This way, I’m learning and will be able to write code on my own for future problems.\nTalk about the Problem Please!! # The challenge for day one was to \u0026ldquo;Predict house prices using Simple Linear Regression\u0026rdquo;. It is a classic problem in machine learning.\nPackages Required. # I installed the necessary packages. Here’s what you need to set up:\npip install pandas scikit-learn matplotlib numpy Why is it a Linear Regression Problem? # It is clearly a regression problem because predicting house prices results in a continuous outcome, not belonging to any set category.\nI chose the Linear Regression model for its simplicity and ease of implementation. Unlike more complex models, it doesn’t require data preprocessing. This makes it an excellent choice for a straightforward Day 1 project.\nUnderstanding the Data # I am using fetch_california_housing from sklearn.datasets. The California Housing dataset is a well-known dataset that contains data about houses in California. It includes various features, but for the simplicity of this example, we\u0026rsquo;ll focus on two key variables:\nMedInc: Median income in the block group\nMedHouseVal: Median house value for California districts (target variable)\nBoston Housing from Kaggle is another excellent option for acquiring a suitable dataset for this problem.\nThe Code Workflow # The workflow involves six major steps:\nLoading the dataset Selecting features and target Splitting the dataset Creating and training the model Evaluating the model\u0026rsquo;s performance Visualizing the results Let\u0026rsquo;s dive into each step:\nStep 1: Load the Dataset # I used fetch_california_housing from sklearn.datasets. I have set the paramter as_frame to true to get the data as a Pandas DataFrame. It will help in analysing the data easily, like with function head(), the table structure with top 5 rows.\ncalifornia_housing = fetch_california_housing(as_frame=True) california_housing_df = california_housing.frame Step 2: Select Features and Target # In Simple Linear Regression, we predict the outcome based on a single feature. Here, I\u0026rsquo;m using median income (MedInc) as our feature stored in X, predicting MedHouseVal as our target y, the median house value.\nStep 3: Split the Dataset # I split the data into a training set (80%) and a validation set (20%).\nStep 4: Create and Train the Model # Create an instance of LinearRegression model and train it using the fit method on the training data.\nStep 5: Evaluation # After training, i have used Root Mean Squared Error (RMSE) to evaluate the accuracy of the model. Here is the result\nThe Root Mean Squared error is: 0.8420901241414454 Step 6: Visualization # I used matplotlib.pyplot package to plot a graph for visualizing the true the true median house values against the predicted values to see how well the model performed.\nGratitude # I finished in 1 hour, which was faster than I planned. I am really happy with this progress and excited to continue the challenge without missing a day.\nStay Tuned!!\nVideo # ","date":"11 September 2024","externalUrl":null,"permalink":"/challenge/ml/30-days-30-ml-projects-day-1/","section":"Challenges","summary":"Today marks Day 1 of my 30 Days, 30 Machine Learning Projects Challenge. The task for today is predicting house prices with Simple Linear Regression. Curious about how it went? Read on to see the results!","title":"Day 1 - 30 Days 30 Machine Learning Projects","type":"challenge"},{"content":"I have been reading a lot about Machine Learning and AI recently and finished a number of tutorials on Coursera, YouTube, and Google. I do understand the basics, but I find myself getting bored too quickly.\nI can read tutorials and all, but they become boring after some time. I believe learning by doing is more fun, especially for an experienced Web Developer like me. So, I have challenged myself to do this challenge of completing 30 small projects in 30 days.\nWhat Projects should I work on? # I want to learn gradually. I don\u0026rsquo;t want to pick a complex project at the start and risk getting stuck. So the plan is to increase the complexity gradually.\nI decided to ask the same question from ChatGPT, and this is what I am planning to follow.\nWeek Day Project 1 1 Predict house prices using Simple Linear Regression 1 2 Classify Iris flowers into species using Logistic Regression 1 3 Recognize handwritten digits with k-Nearest Neighbors on MNIST 1 4 Diagnose breast cancer as malignant or benign using a Decision Tree 1 5 Filter spam from a collection of emails using Naive Bayes 1 6 Predict wine quality from physicochemical properties using SVM 1 7 Determine Credit Card defaults using a Random Forest Classifier 2 8 Detecting fake news with a PassiveAggressive Classifier and TfidfVectorizer 2 9 Forecasting weather with Simple Linear Regression on time series data 2 10 Recommender System using Collaborative Filtering on user-item ratings matrix 2 11 Anomaly detection in network traffic with Isolation Forest 2 12 Predicting airline passenger satisfaction with Gradient Boosting Machine (GBM) 2 13 Build a music genre classifier using audio features extraction 2 14 Cluster grocery store customers based on purchase history with K-Means 3 15 Predict house prices with XGBoost 3 16 Real-time face detection in a webcam feed using OpenCV 3 17 Predict diabetes onset using Decision Trees and Random Forests 3 18 Time Series Forecasting of stock prices with ARIMA model 3 19 Customer churn prediction with XGBoost 3 20 Create a topic model using Latent Dirichlet Allocation (LDA) 4 21 Deploy a machine learning model using FastAPI and Heroku for real-time predictions 4 22 Recommender System with Matrix Factorization 4 23 Fraud Detection in Financial Transactions using Logistic Regression and Random Forest 4 24 K-Means clustering to segment customers based on behavior 4 25 Sentiment Analysis of customer reviews using traditional NLP techniques 4 26 Time Series Forecasting of electricity consumption using LSTM (Deep Learning Intro) 4 27 Image Classification with a small CNN on CIFAR-10 dataset 4 28 Build a simple chatbot using traditional NLP techniques 5 29 Credit risk prediction with Logistic Regression and SVM 5 30 Capstone Project: Predicting loan approvals using ensemble learning (Random Forest, XGBoost) The Logic Behind Choosing These Problems # It is structured to help build a solid foundation, gradually move towards advanced machine learning topics, and then introduce you to deep learning concepts in a way that feels more natural. Here’s why I selected these specific problems:\nWeek 1: Core Supervised Learning Concepts # Days 1-7: These are foundational tasks designed to help me understand the basic principles of regression and classification. I explore key algorithms (Linear Regression, Logistic Regression, k-NN, Decision Trees, Naive Bayes, SVM, Random Forest) through hands-on tasks:\nLinear Regression: A simple start with predicting house prices, helping me understand the essence of regression. Logistic Regression: Moving to binary classification with the Iris dataset, introducing me to probability-based classifications. k-NN \u0026amp; MNIST: Here, I dive into distance-based learning, setting the stage for image classification tasks later on. Decision Tree \u0026amp; Naive Bayes: Both methods offer different perspectives on handling structured classification tasks. SVM for Wine Quality: A more abstract but powerful introduction to hyperplanes and margins in classification. Random Forest: Brings in ensemble learning, introducing the idea of combining weak learners for stronger predictions. Week 2: Applying Machine Learning to Real-World Problems # Days 8-14: Now that I’ve grasped the basics, I’ll apply machine learning to real-world datasets, which makes the learning more relevant:\nFake News Detection: A dive into NLP with the TfidfVectorizer to detect fake news—a highly practical application. Weather Forecasting: Time-series forecasting deepens my understanding of regression and pattern recognition. Recommender Systems: By exploring collaborative filtering, I’m learning how recommendation engines work in real-world applications. Anomaly Detection: Isolation Forest introduces unsupervised learning, focusing on identifying anomalies in data. Gradient Boosting (GBM): I take ensemble learning a step further with GBM to boost prediction accuracy. Music Genre Classification: A fun shift to working with audio features, transitioning away from structured and text data. Clustering: A practical use of clustering for segmenting customers, setting me up for more unsupervised tasks later on. Week 3: Introducing More Advanced Techniques # Days 15-21: Time to dive into more complex algorithms and techniques that build on my previous learning:\nXGBoost for House Prices: To see how XGBoost outshines GBM and Random Forest by better handling overfitting and providing more control. Real-Time Face Detection: Using OpenCV introduces me to computer vision, but I’ll hold off on deep learning for now. Diabetes Prediction: I revisit classification, applying decision trees and random forests for more practical, medical predictions. Stock Price Forecasting with ARIMA: This prepares me for time-series forecasting with recurrent neural networks (RNNs) in the future. Customer Churn Prediction: A crucial business task, I’ll learn how to predict customer retention using XGBoost. Topic Modeling with LDA: I dive deeper into NLP by understanding how unsupervised learning can extract hidden patterns from text. Model Deployment: I bring it all together by deploying my models with Flask and Heroku, understanding how to make them live and usable. Week 4: Prepping for Deep Learning and Advanced Topics # Days 22-30: By now, I’m ready to either dive into deep learning or explore more advanced models. This week will help me transition to the next level:\nMatrix Factorization for Recommendations: I’ll build a more advanced recommender system using matrix factorization. Financial Fraud Detection: Applying ensemble learning to a real-world problem with significant business impact. Customer Segmentation with K-Means: I deepen my clustering skills, working on a real-world marketing problem. Sentiment Analysis with traditional NLP: I revisit NLP to solidify my text analysis techniques before diving into deep learning. LSTM for Time Series Forecasting: Finally, I step into deep learning with LSTM for time series, opening up the world of RNNs. Image Classification with CNN: My first attempt at building a Convolutional Neural Network (CNN) using CIFAR-10, a major milestone in deep learning for image data. Building a Chatbot: This practical NLP problem helps me understand how businesses are utilizing machine learning for interactive tasks. Prediction with Logistic Regression and SVM: An advanced take on blending two powerful classification models. Capstone Project: ’ll wrap up with an advanced project, bringing multiple concepts together (like ensemble learning) to predict loan approvals, acting as a final showcase of everything I’ve learned. Why This Structure Works: # Gradual Introduction of Complexity: I started the challenge with easier problems and models, but each week I’m adding layers of complexity. By week 3, I’m ready to take on more complex algorithms and tasks.\nUnsupervised Learning: I’m introducing techniques like clustering in week 2, but I’ll return to them with more advanced datasets and models in weeks 3 and 4. Deep Learning Gradual Introduction: Instead of diving straight into deep learning, I’m starting with traditional methods (week 3) and only moving into deep learning models like LSTM and CNN toward the end of the challenge. Real-World Application: Every task I’m working on is designed to solve a real-world problem, whether it’s in finance, marketing, image recognition, or NLP, ensuring it’s always relevant. With this structure, I’ll ease into more advanced topics like deep learning, so I won’t feel overwhelmed like before!\nPlan # I am a morning person. I am planning to dedicate 2 hours in the early morning for the project and write up the progress post in the evening after my work hours.\nThe challenge starts on the 11th of September 2024 and ends on the 10th of October 2024.\nGit repository for the code-base: \\[HERE\\](https://github.com/saxenaakansha30/30-days-ml-challenge)\nWhy am I sharing this? # I saw myself procrastinating a lot lately. So, posting my updates here will leave me accountable. Solving every day, I may not end up with the perfect solution for the projects, but the learning and iterative development process is the key to learning any new skill.\nCome and join if you feel the same!! Otherwise too ;)\n","date":"10 September 2024","externalUrl":null,"permalink":"/post/30-days-30-ml-projects-challenge/","section":"Post","summary":"I’ve been learning a lot about AI and machine learning from online courses. But reading tutorials is getting boring, even though I know the basics. As a web developer, I like to learn by doing. So, I have challenged myself to do this challenge of completing 30 small projects in 30 days. Come join me on this journey!","title":"30 Days, 30 Machine Learning Projects","type":"post"},{"content":"","date":"10 September 2024","externalUrl":null,"permalink":"/challenge/ml/","section":"Challenges","summary":"List of Problems for 30 Days, 30 Machine Learning Projects Challenge","title":"30 Days, 30 Machine Learning Projects List","type":"challenge"},{"content":"","date":"6 August 2024","externalUrl":null,"permalink":"/tags/drupal/","section":"Tags","summary":"","title":"Drupal","type":"tags"},{"content":"Welcome back! After our first article that showed you our new Drupal Rag Integration app, many of you liked it. Thank you! Today, let\u0026rsquo;s look at how the code works.\nQuick Reminder # If you missed our first article or need a quick reminder, you can take a look at it here. It\u0026rsquo;ll help you understand what we\u0026rsquo;re talking about today.\nHow the App is Built # Our integration revolves around a dynamic interaction between a backend designed for intelligent data retrieval and augmentation, and a robust Content Management System (CMS) frontend.\nRAG Backend:\nVector Database: Chroma Rag Backend Framework: FastAPI with Python Local Language Model Abstraction Layer (OLLAMA): Introduces a layer that allows for local language model processing. LLM Model: Mistral Programming Language: Python 3.6 Website:\nCMS: Drupal 10 Database: MySQL Programming Language: PHP 8.1 Key APIs of the Integration # The heart of our integration lies within these four APIs:\nAdd Feed API (/feed/add): # Method: POST Parameters: node_id (string), data (string) Returns: document_ids (list of strings) Description: Content is divided into smaller chunks via RecursiveCharacterTextSplitter, assigning unique IDs to each chunk. These IDs play a vital role in future update and delete operations. Subsequently, chunks are stored with distinct document_ids in the Vector Database. @app.post(\u0026#34;/feed/add\u0026#34;) def feed_add(feed_data: FeedData = Body(...)): nid = feed_data.nid data = feed_data.data ids = add_docs(nid=nid, data=data) # Return ids return {\u0026#34;response\u0026#34;: \u0026#34;Document successfully added.\u0026#34;, \u0026#34;doc_ids\u0026#34;: ids} Update Feed API (/feed/update): # Method: POST Parameters: node_id (string), document_ids (list of strings), data (string) Returns: document_ids (array of strings) Description: Using the given document_ids, existing documents are deleted. Identical to the add_feed process, this API repopulates the database with updated information, returning new document_ids. @app.post(\u0026#34;/feed/update\u0026#34;) def feed_update(feed_data: UpdateData = Body(...)): nid = feed_data.nid ids = feed_data.ids data = feed_data.data vectordb_manager = VectorDbManager() # Delete with ids passed. vectordb_manager.delete_ids(ids=ids) # Create fresh documents. new_ids = add_docs(nid=nid, data=data) # return the ids return {\u0026#34;response\u0026#34;: \u0026#34;Document successfully updated.\u0026#34;, \u0026#34;doc_ids\u0026#34;: new_ids} Delete Feed API (/feed/delete): # Method: DELETE Parameters: document_ids (array of strings) Returns: HTTP status 200 if successfully deleted Description: Removes documents using their unique identifiers via a Chroma delete query. @app.post(\u0026#34;/feed/delete\u0026#34;) def feed_delete(data: DeleteData = Body(...)): ids = data.ids # Delete with ids passed. vectordb_manager = VectorDbManager() vectordb_manager.delete_ids(ids=ids) return {\u0026#34;response\u0026#34;: \u0026#34;Document successfully deleted.\u0026#34;} Chroma Vector Database # Configurations for the Chroma Vector Database reside in the vector_manager.py under the VectorDbManager.store_data() method, with the default storage location being /chroma_data. This location is pivotal and can be adjusted as per requirement.\nvectordb = chroma.Chroma.from_documents( documents=chunks, embedding=fastembed.FastEmbedEmbeddings(), persist_directory=\u0026#34;chroma_data\u0026#34;, ids=ids ) vectordb.persist() Drupal Integration Schema and Hooks # We introduce a schema identified as drupal_rag_integration_node_doc to bridge node_id and document_ids. Drupal\u0026rsquo;s hooks hook_ENTITY_TYPE_insert, hook_ENTITY_TYPE_update, and hook_ENTITY_TYPE_delete operate in tandem with these APIs to manage the node\u0026rsquo;s lifecycle through JSON-formatted data packets.\n/** * Implements hook_ENTITY_TYPE_insert() for node entities. * * @param \\Drupal\\Core\\Entity\\EntityInterface $entity * The node entity that was inserted. */ function drupal_rag_integration_node_insert(EntityInterface $entity) { /** @var RagEntityOperations $entity_operations */ $entity_operations = \\Drupal::service(\u0026#39;drupal_rag_integration.entity_operations\u0026#39;); $entity_operations-\u0026gt;handleInsert($entity); } /** * Implements hook_ENTITY_TYPE_update() for node entities. * * @param \\Drupal\\Core\\Entity\\EntityInterface $entity * The node entity that was updated. */ function drupal_rag_integration_node_update(EntityInterface $entity) { /** @var RagEntityOperations $entity_operations */ $entity_operations = \\Drupal::service(\u0026#39;drupal_rag_integration.entity_operations\u0026#39;); $entity_operations-\u0026gt;handleUpdate($entity); } /** * Implements hook_ENTITY_TYPE_delete() for node entities. * * @param \\Drupal\\Core\\Entity\\EntityInterface $entity * The node entity that was deleted. */ function drupal_rag_integration_node_delete(EntityInterface $entity) { /** @var RagEntityOperations $entity_operations */ $entity_operations = \\Drupal::service(\u0026#39;drupal_rag_integration.entity_operations\u0026#39;); $entity_operations-\u0026gt;handleDelete($entity); } User Interaction Through ASK Form # At the frontend, users engage a form aptly named \u0026ldquo;ASK\u0026rdquo; to query the Drupal database and other general inquiries. The form leverages the Ask API for this purpose.\npublic function submitForm(array \u0026amp;$form, FormStateInterface $form_state): void { $question = $form_state-\u0026gt;getValue(\u0026#39;question\u0026#39;); $endpoint = \u0026#39;/ask\u0026#39;; $payload = json_encode([\u0026#39;question\u0026#39; =\u0026gt; $question]); $response = $this-\u0026gt;apiClient-\u0026gt;callApi($endpoint, $payload); if (isset($response[\u0026#39;response\u0026#39;])) { $form_state-\u0026gt;set(\u0026#39;response\u0026#39;, $response[\u0026#39;response\u0026#39;]); } else { $form_state-\u0026gt;set(\u0026#39;response\u0026#39;, $this-\u0026gt;t(\u0026#39;Error occurred: @error\u0026#39;, [\u0026#39;@error\u0026#39; =\u0026gt; $response[\u0026#39;error\u0026#39;] ?? $this-\u0026gt;t(\u0026#39;Unknown error\u0026#39;)])); } $form_state-\u0026gt;setRebuild(TRUE); } Ask API (/ask) # Method: POST Parameters: question (string) Returns: Generated response (string) Description: Implements context retrieval from user prompts via the Chroma Vector Database, subsequently passing the augmented query to the Mistral LLM model, which formulates the final response. @app.post(\u0026#34;/ask\u0026#34;) def ask(data: Question = Body(...)): question = data.question rag_obj = Rag() rag_obj.set_retrieve() rag_obj.augment() response = rag_obj.generate(question) return {\u0026#34;response\u0026#34;: response} Watch the Detailed Video Explanation # Hope the article helps in some way. Stay tuned for more articles like this!\n","date":"6 August 2024","externalUrl":null,"permalink":"/post/drupal-rag-integration-code-explained/","section":"Post","summary":"Welcome back! After our first article that showed you our new Drupal Rag Integration app, many of you liked it. Thank you! Today, let’s look at how the code works.","title":"Inside the Codebase: A Deep Dive into Drupal Rag Integration","type":"post"},{"content":"","date":"6 August 2024","externalUrl":null,"permalink":"/categories/ollama/","section":"Categories","summary":"","title":"Ollama","type":"categories"},{"content":"","date":"6 August 2024","externalUrl":null,"permalink":"/tags/ollama/","section":"Tags","summary":"","title":"Ollama","type":"tags"},{"content":"","date":"6 August 2024","externalUrl":null,"permalink":"/categories/rag/","section":"Categories","summary":"","title":"Rag","type":"categories"},{"content":"","date":"6 August 2024","externalUrl":null,"permalink":"/series/rag/","section":"Series","summary":"","title":"Rag","type":"series"},{"content":"","date":"6 August 2024","externalUrl":null,"permalink":"/tags/rag/","section":"Tags","summary":"","title":"Rag","type":"tags"},{"content":"","date":"6 August 2024","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"},{"content":"As a Drupal developer who values open-source solutions, I was excited to explore the potential of combining Drupal and Ollama. Drupal is one of the most popular CMS platforms worldwide. Tesla, Nokia, and Oxford University are just a few examples of high-traffic websites powered by Drupal. Among the many reasons why Drupal is used by so many companies, the core one that resonates with me is its open-source nature.\nOllama is also open-source and is being used extensively to create applications around LLMs.\nI have used these two powerful tools to create an application called Drupal RAG Integration.\nLet\u0026rsquo;s understand the use case # Imagine you run a tech blog on Drupal. A visitor asks, \u0026lsquo;What was your latest article about AI?\u0026rsquo; Your chatbot can now provide an accurate, up-to-date response based on your actual content.\nIntegration with OpenAI or any other LLM alone is not sufficient. While they can answer general questions they\u0026rsquo;ve been trained on, they can\u0026rsquo;t handle content specific to your website. One option is to train the LLMs, but that is too expensive and requires machine learning expertise. With RAG, we can implement this functionality easily.\nIn Retrieval Augmented Generation (RAG) architecture, we can retrieve additional context from our custom data source and pass it to general-purpose LLMs to generate personalized and accurate responses.\nAbout the Application # This integration empowers Drupal site owners to create intelligent, content-aware chatbots without the need for expensive AI training or external services.\nAll Drupal content, when created, updated, or deleted, is stored in the Chroma vector store and will be used later to retrieve additional context for the LLM to generate responses.\nI have used FastAPI to provide APIs for interacting with LLMs and storing data in the Chroma vector storage.\nOn the Drupal side, I\u0026rsquo;ve built a module called drupal_rag_integration that feeds data to the RAG app backend and provides a form for users to ask questions and receive generated responses.\nArchitecture # To better understand how Drupal RAG Integration works, let\u0026rsquo;s take a look at its high level architecture.\nThe diagram above illustrates the flow of data from Drupal content creation to the Chroma vector store, and how queries are processed through the RAG system.\nDemo # In this short video, you\u0026rsquo;ll see how quickly the chatbot responds with accurate, site-specific information.\nCodebase # The code for application is available on my github.\nDrupal module: https://github.com/saxenaakansha30/drupal-rag-integration-module\nRag FastAPI codebase: https://github.com/saxenaakansha30/drupal-rag-app\nExplain please!! # In the next article, we\u0026rsquo;ll dive deeper into the technical details.\nStay tuned!!\n","date":"29 July 2024","externalUrl":null,"permalink":"/post/drupal-rag-integration/","section":"Post","summary":"As a Drupal developer who values open-source solutions, I was excited to explore the potential of combining Drupal and Ollama. Drupal is one of the most popular CMS platforms worldwide. Tesla, Nokia, and Oxford University are just a few examples of high-traffic websites powered by Drupal. Among the many reasons why Drupal is used by so many companies, the core one that resonates with me is its open-source nature.","title":"Build Smart Drupal Chatbots with RAG Integration and Ollama","type":"post"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/tags/chroma/","section":"Tags","summary":"","title":"Chroma","type":"tags"},{"content":"In this article, we build a Retrieval-Augmented Generation (RAG) web application called DocuMentor that allows users to upload PDF documents and ask questions about the contents.\nLet\u0026rsquo;s understand this with the help of a real-world example.\nLet\u0026rsquo;s say you have a web app for your hospital that contains extensive data about doctors—their names, departments, phone numbers, working hours, and more. Now you want to build a chatbot like ChatGPT, but it should address specific queries about your data, such as the details of doctors working at your hospital.\nHere\u0026rsquo;s how it would look:\nNote: I have used a PDF that is publicly available on the official website of the Uttar Pradesh National Health Mission.\nWithout wasting any more time, let\u0026rsquo;s get started.\nDependencies # langchain streamlit streamlit_chat pypdf chromadb fastembed pip install langchain langchain_community streamlit streamlit_chat chromadb pypdf fastembed Tech stack used: # We will use\nOllama for running the LLMs locally. If you want to learn more about Ollama and how to get started with it locally, visit this article first. Lllama Model mistral: Use ollama list to check if it is installed on your system, or else use the command ollama pull mistral to download the latest default manifest of the model. Chroma vector database to store the PDF document\u0026rsquo;s vector embeddings. Streamlit to build the UI of the application. If the jargon used here sounds alien to you, I recommend referring to the following blogs to understand the RAG concept and learn how to build a RAG application from scratch without any external vector database.\nRetrieval Augmented Generation (RAG): A Beginner’s Guide to This Complex Architecture. Song Recommender: Building a RAG Application for Beginners From Scratch Full Code # The full code of the application is available on the Git repository https://github.com/saxenaakansha30/documentor\nLet\u0026rsquo;s Code # We will have 3 files.\nmain.py: Implements UI of the app. rag.py: Implements retrieval, augmentation and response generation. chunk_vector_store.py: Implements class to split PDF into chunks and provide vector store. File main.py # import streamlit as st import tempfile import os # Import Rag classes. from rag import Rag #Display all messages stored in session_state def display_messages(): for message in st.session_state.messages: with st.chat_message(message[\u0026#39;role\u0026#39;]): st.markdown(message[\u0026#39;content\u0026#39;]) def process_file(): st.session_state[\u0026#34;assistant\u0026#34;].clear() st.session_state.messages = [] for file in st.session_state[\u0026#34;file_uploader\u0026#34;]: # Store the file at tem location # of your system to feed to our vector storage. with tempfile.NamedTemporaryFile(delete=False) as tf: tf.write(file.getbuffer()) file_path = tf.name #feed the file to the vector storage. with st.session_state[\u0026#34;feeder_spinner\u0026#34;], st.spinner(\u0026#34;Uploading the file\u0026#34;): st.session_state[\u0026#34;assistant\u0026#34;].feed(file_path) os.remove(file_path) def process_input(): # See if user has typed in any message and assign to prompt. if prompt := st.chat_input(\u0026#34;What can i do?\u0026#34;): with st.chat_message(\u0026#34;user\u0026#34;): st.markdown(prompt) st.session_state.messages.append({\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: prompt}) # Generate response and write back to the chat container. response = st.session_state[\u0026#34;assistant\u0026#34;].ask(prompt) with st.chat_message(\u0026#34;assistant\u0026#34;): st.markdown(response) st.session_state.messages.append({\u0026#34;role\u0026#34;: \u0026#34;assistant\u0026#34;, \u0026#34;content\u0026#34;: response}) def main(): st.title(\u0026#34;DocueMentor\u0026#34;) # Initialize the session_state. if len(st.session_state) == 0: st.session_state[\u0026#34;assistant\u0026#34;] = Rag() st.session_state.messages = [] # Code for file upload functionality. st.file_uploader( \u0026#34;Upload the document\u0026#34;, type = [\u0026#34;pdf\u0026#34;], key = \u0026#34;file_uploader\u0026#34;, on_change=process_file, label_visibility=\u0026#34;collapsed\u0026#34;, accept_multiple_files=True, ) st.session_state[\u0026#34;feeder_spinner\u0026#34;] = st.empty() display_messages() process_input() if __name__ == \u0026#34;__main__\u0026#34;: main() Streamlit\u0026rsquo;s official documentation explains very nicely how to build a chatbot UI. Refer to the article Build chatbot UI using streamlit\nWe have two important functions: feed(file_path) and ask(prompt). The feed function is used to feed the PDF file into our Chroma database, and ask is the callback to the user_input. It returns a response by first retrieving the context from the PDF with the maximum similarity stored in the Chroma vector database, then asking the Mistral LLM for a response with this additional context.\nFile rag.py implements class Rag # from chunk_vector_store import ChunkVectorStore as cvs from langchain.schema.runnable import RunnablePassthrough from langchain.schema.output_parser import StrOutputParser from langchain.prompts import PromptTemplate from langchain_community.chat_models import ChatOllama class Rag: vector_store = None retriever = None chain = None def __init__(self) -\u0026gt; None: self.csv_obj = cvs() self.prompt = PromptTemplate.from_template( \u0026#34;\u0026#34;\u0026#34; \u0026lt;s\u0026gt; [INST] You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don\u0026#39;t know the answer, just say that you don\u0026#39;t know. Use three sentences maximum and keep the answer concise. [/INST] \u0026lt;/s\u0026gt; [INST] Question: {question} Context: {context} Answer: [/INST] \u0026#34;\u0026#34;\u0026#34; ) self.model = ChatOllama(model=\u0026#34;mistral\u0026#34;) def set_retriever(self): self.retriever = self.vector_store.as_retriever( search_type=\u0026#34;similarity_score_threshold\u0026#34;, search_kwargs={ \u0026#34;k\u0026#34;: 3, \u0026#34;score_threshold\u0026#34;: 0.5, }, ) # Augment the context to original prompt. def augment(self): self.chain = ({\u0026#34;context\u0026#34;: self.retriever, \u0026#34;question\u0026#34;: RunnablePassthrough()} | self.prompt | self.model | StrOutputParser()) # Generate the response. def ask(self, query: str): if not self.chain: return \u0026#34;Please upload a PDF file for context\u0026#34; return self.chain.invoke(query) # Stores the file into vector database. def feed(self, file_path: str): chunks = self.csv_obj.split_into_chunks(file_path) self.vector_store = self.csv_obj.store_to_vector_database(chunks) self.set_retriever() self.augment() def clear(self): self.vector_store = None self.chain = None self.retriever = None chunk_vector_store file implementing ChunkVectorStore class # from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.vectorstores.utils import filter_complex_metadata from langchain_community.document_loaders import PyPDFLoader from langchain_community.vectorstores import chroma; from langchain_community.embeddings import fastembed; class ChunkVectorStore: def __init__(self) -\u0026gt; None: pass def split_into_chunks(self, file_path: str): doc = PyPDFLoader(file_path).load() text_splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=20) chunks = text_splitter.split_documents(doc) chunks = filter_complex_metadata(chunks) return chunks def store_to_vector_database(self, chunks): return chroma.Chroma.from_documents(documents=chunks, embedding=fastembed.FastEmbedEmbeddings()) We use 3 public and 3 local class variables. Let\u0026rsquo;s understand them:\nvector_store: is a Chroma vector database object initialized with chunks and the type of embedding used to convert those chunks. retriever: An object of vector_store used as retriever. It retrieves the top 3 elements with a similarity threshold \u0026gt;= 0.5 using the similarity_score_threshold search technique. csv_obj: Object of the ChunkVectorStore class. prompt: A prompt template to be filled later with context and question using the Python chaining technique. model: Object for invoking the Mistral LLM model. chain: Used for chaining. In summary,\nWhen a user uploads a PDF document, process_file() in main.py is triggered. It uploads the file to the system\u0026rsquo;s temporary location using the tempfile module. The file path is then passed to the feed() method of the Rag class, stored as an object in Streamlit\u0026rsquo;s session_state[\u0026ldquo;assistant\u0026rdquo;].\nfeed() uses:\nChunkVectorStore to split the PDF into chunks of size 1024 bytes using RecursiveCharacterTextSplitter. It then calls ChunkVectorStore.store_to_vector_database() to create and the return vector_store The vector store is then set as a retriever by calling the function Rag.set_retriever(). Once the retriever is ready, we prepare the prompt template by calling Rag.augment(). When a user asks a question, it\u0026rsquo;s passed to the ask() method of the Rag class. The function checks if the retriever is set, along with other variables, by using a chaining technique stored in the variable chain. If everything is configured, it invokes the Mistral model with the prompt and context and returns the response.\nCongratulations! We have successfully built a functional RAG application.\nVideo Explaination # If you want video version of this content, check out this video.\n","date":"1 July 2024","externalUrl":null,"permalink":"/post/documentor-rag-app/","section":"Post","summary":"In this article, we build a Retrieval-Augmented Generation (RAG) web application called DocuMentor that allows users to upload PDF documents and ask questions about the contents.\nLet’s understand this with the help of a real-world example","title":"DocuMentor: Build a RAG Chatbot with Ollama, Chroma \u0026 Streamlit","type":"post"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/categories/vector-database/","section":"Categories","summary":"","title":"Vector-Database","type":"categories"},{"content":"","date":"1 July 2024","externalUrl":null,"permalink":"/tags/vector-database/","section":"Tags","summary":"","title":"Vector-Database","type":"tags"},{"content":"Have you been trying to understand RAG and read a bunch of articles, yet feel overwhelmed by how complicated it seems to implement? You\u0026rsquo;ve just hit the jackpot. In this article, I aim to demystify the RAG concept and demonstrate its utility by building a hands-on song recommender application – all without unnecessary complexity. There\u0026rsquo;s no need for a deep understanding of AI and machine learning; basic knowledge of Python and a willingness to learn something new are all that\u0026rsquo;s required. So, without further ado, let\u0026rsquo;s get started.\nTo know more about RAG - Read my preious article here\nImagine this: you\u0026rsquo;re humming a tune or a string of lyrics is stuck in your head. You loved the song and want to stay in the groove by listening to similar songs. That\u0026rsquo;s exactly where our RAG application comes into play. Just type in the lyrics or mention the genre, and voila! Our application recommends a song that aligns with your input.\nLet\u0026rsquo;s build a small database of songs with their titles and details, where \u0026lsquo;detail\u0026rsquo; includes a few lines from the lyrics of the song and its genre.\nsongs_corpus = [ {\u0026#34;title\u0026#34;: \u0026#34;Bohemian Rhapsody\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Is this the real life?; genre: Roc\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Shake It Off\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Players gonna play, hate; genre: Pop\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Thriller\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Cause this is thriller; genre: Pop\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Rolling in the Deep\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: There\u0026#39;s a fire start; genre: Pop\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Smells Like Teen Spirit\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: With the lights out; genre: Gru\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Hotel California\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: On a dark desert hwy; genre: Roc\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Sweet Child o\u0026#39; Mine\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: She\u0026#39;s got eyes blue; genre: Roc\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Wonderwall\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Because maybe, save; genre: Alt\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Billie Jean\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: But the kid is not; genre: Pop\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Firework\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Do you ever feel so; genre: Pop\u0026#34;} ] To find the similarity between the user input and the songs of the corpus, we are going to use Jaccard Similarity.\nlet\u0026rsquo;s understand with an example:\nSet A: {\u0026ldquo;I\u0026rdquo;, \u0026ldquo;love\u0026rdquo;, \u0026ldquo;cats\u0026rdquo;, \u0026ldquo;and\u0026rdquo;, \u0026ldquo;dogs\u0026rdquo;}\nSet B: {\u0026ldquo;We\u0026rdquo;, \u0026ldquo;love\u0026rdquo;, \u0026ldquo;cats\u0026rdquo;, \u0026ldquo;not\u0026rdquo;, \u0026ldquo;dogs\u0026rdquo;}\nJaccard Similarity = Intersection(A, B) / Union(A, B)\nIntersection(A, B) = common words in A and B = love, cats, dogs = 3 words\nUnion(A, B) = All unique words in A and B = I, love, cats, and, dogs, we, not = 7 words\nJaccard Similarity = 3 / 7 = 0.42\nLet\u0026rsquo;s code it up.\ndef tokenize(text): return set(text.lower().split(\u0026#34; \u0026#34;)) # Mesaures similarity between two data sets. # Jaccard Index = Intersection (A, B) / Union (A, B) def jaccard_similarity(query, document): tokenize_query = tokenize(query) tokenize_document = tokenize(document) intersection = tokenize_query.intersection(tokenize_document) union = tokenize_query.union(tokenize_document) similarity = len(intersection) / len(union) return similarity Now use the jaccard_similarity function to get the song with maximum similarity from our songs corpus.\ndef get_relevant_document(query): relavant_song_title = \u0026#39;\u0026#39; max_similarity = 0 for song in songs_corpus: detail = song[\u0026#39;detail\u0026#39;] similarity = jaccard_similarity(query, detail) if similarity \u0026gt; max_similarity: max_similarity = similarity relavant_song_title = song[\u0026#39;title\u0026#39;] return relavant_song_title Let\u0026rsquo;s test what we have built so far.\nuser_input = input(\u0026#34;Tell me what are you thinking, i will recommand a song.\\n\u0026#34;) relevant_document = get_relevant_document(user_input) print(\u0026#34;I reccomed you to listen: \u0026#34; + relevant_document) Output:\nCongratulations, it works! This process of retrieving the relevant data from your own content is known as Retrieval. Now, let\u0026rsquo;s test one more example.\nOutput:\nIt did not work with the negative case. That\u0026rsquo;s where we need a large language model to fallback on.\nItegrate LLM: # Augmentation: We will augment this relevant song with the original user input and prepare a prompt to pass to the LLM for generating a response.\n#Augment the relavant song to original query. prompt = f\u0026#39;\u0026#39;\u0026#39; You are a bot that makes recommendations for songs. You answer in very short sentences and do not include extra information. This is the recommended song: {relevant_document} The user input is: {user_input} Compile a recommendation to the user based on the recommended song and the user input. If the user has no interset simple deny. \u0026#39;\u0026#39;\u0026#39; Generation\nThis step is pretty simple: pass the prompt we have generated to the LLM. We are using Ollama to run the LLM (llama3) locally, and Langchain to invoke it.\nMore about ollama and how to run at local, LINK\nLangchain: Langchain is a framework designed to simplify the creation of applications using large language models. Learn more about Langchain.\nThe code is straightforward: import the LLM package from langchain_community and use its Ollama() method to specify which LLM model you want to use for your application. I am using llama3:latest. Once you have created the llm object, use the invoke method to pass the prompt and invoke the LLM to generate the response. We wrap all of this in the get_response() function.\nfrom langchain_community.llms import ollama def get_response(prompt): llm = ollama.Ollama(model=\u0026#34;llama3:latest\u0026#34;) response = llm.invoke(prompt) return response Final touch: call the get_response and print to the user.\nresponse = get_response(prompt) print(\u0026#34;I reccomed you to listen: \u0026#34; + response) Test it on the same inputs.\nAttempt 1: Attempt 1: Congratulations! You have built a RAG application from scratch.\nIn the next article, we will delve a little deeper. Stay tuned!\nFull code\nfrom langchain_community.llms import ollama songs_corpus = [ {\u0026#34;title\u0026#34;: \u0026#34;Bohemian Rhapsody\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Is this the real life?; genre: Roc\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Shake It Off\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Players gonna play, hate; genre: Pop\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Thriller\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Cause this is thriller; genre: Pop\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Rolling in the Deep\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: There\u0026#39;s a fire start; genre: Pop\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Smells Like Teen Spirit\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: With the lights out; genre: Gru\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Hotel California\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: On a dark desert hwy; genre: Roc\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Sweet Child o\u0026#39; Mine\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: She\u0026#39;s got eyes blue; genre: Roc\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Wonderwall\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Because maybe, save; genre: Alt\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Billie Jean\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: But the kid is not; genre: Pop\u0026#34;}, {\u0026#34;title\u0026#34;: \u0026#34;Firework\u0026#34;, \u0026#34;detail\u0026#34;: \u0026#34;lyrics: Do you ever feel so; genre: Pop\u0026#34;} ] def tokenize(text): return set(text.lower().split(\u0026#34; \u0026#34;)) # Mesaures similarity between two data sets. # Jaccard Index = Intersection (A, B) / Union (A, B) def jaccard_similarity(query, document): tokenize_query = tokenize(query) tokenize_document = tokenize(document) intersection = tokenize_query.intersection(tokenize_document) union = tokenize_query.union(tokenize_document) similarity = len(intersection) / len(union) return similarity def get_relevant_document(query): relavant_song_title = \u0026#39;\u0026#39; max_similarity = 0 for song in songs_corpus: detail = song[\u0026#39;detail\u0026#39;] similarity = jaccard_similarity(query, detail) if similarity \u0026gt; max_similarity: max_similarity = similarity relavant_song_title = song[\u0026#39;title\u0026#39;] return relavant_song_title user_input = input(\u0026#34;Tell me what are you thinking, i will recommand a song.\\n\u0026#34;) relevant_document = get_relevant_document(user_input) #Augment the relavant song to original query. prompt = f\u0026#39;\u0026#39;\u0026#39; You are a bot that makes recommendations for songs. You answer in very short sentences and do not include extra information. This is the recommended song: {relevant_document} The user input is: {user_input} Compile a recommendation to the user based on the recommended song and the user input. If the user has no interset simple deny. \u0026#39;\u0026#39;\u0026#39; def get_response(prompt): llm = ollama.Ollama(model=\u0026#34;llama3:latest\u0026#34;) response = llm.invoke(prompt) return response response = get_response(prompt) print(\u0026#34;I reccomed you to listen: \u0026#34; + response) Github Repository # You can find the github repository here\nVideo Explaination # ","date":"6 June 2024","externalUrl":null,"permalink":"/post/song-recommander-rag-app/","section":"Post","summary":"Have you been trying to understand RAG and read a bunch of articles, yet feel overwhelmed by how complicated it seems to implement? You’ve just hit the jackpot. In this article, I aim to demystify the RAG concept and demonstrate its utility by building a hands-on song recommender application – all without unnecessary complexity. There’s no need for a deep understanding of AI and machine learning; basic knowledge of Python and a willingness to learn something new are all that’s required. So, without further ado, let’s get started.","title":"Song Recommender: Building a RAG Application for Beginners from Scratch","type":"post"},{"content":"RAG, or Retrieval Augmented Generation, is getting a lot of talk these days. It\u0026rsquo;s pretty exciting, but also a bit confusing for beginners. In this article I attempt to explain RAG in a way that\u0026rsquo;s easier to understand, without all the complex tech words.\nWhat is RAG? # RAG stands for Retrieval Augmented Generation, an architecture that lets you feed your own content to a generic Large Language Model (LLM) to generate relevant responses.\nThink of the following scenarios:\nExample 1: Easy to understand example: # You have to write an essay on History of India. The first thing you do is to use search engine to look up the information. You find the links to the articles/documentation that contain the information you need and then you write the essay in your own words based of that information.\nRag works in the same way. It is divided into two components. Retriever and Generation. So if I categorise the essay example into Retriever and Generation.\nThe job of Retriever is to get relevant content from the internet in the form of articles/document. And Generation will be to use it to write essay in your own words.\nExample 2: A little technical use case. # You have a support system and you are thinking of using LLM to create a chatbot to help with the questions and answer. Now let\u0026rsquo;s see how this bot works with or without RAG.\nA user comes and asks- I am seeing a white screen of death on this page. How do I resolve it. LLM response: A generic response. Might or might be related to your system.\nChatbot before RAG:\nWith RAG:\nYou can use your content to provide relevant response around your system to the user. You can use old solutions as the data.\nUser —\u0026gt; Old Response —\u0026gt; Augment to the original Prompt -\u0026gt; LLM Benefits: # Improved Accuracy with factually correct responses to queries. Avoid hallucinations or irrelevant responses Access to up-to-date information and reduced data training times since RAG models aren\u0026rsquo;t limited to the data they were initially trained on How does it work: # User: User sends the prompt or can say ask a question.\nRetrieval: # It is a process of finding relevant information from a large dataset that can help in generating accurate responses to queries. This is typically achieved through the following process:\nIndexing Knowledge: Split the data in small chunks. Depending on the nature of the data, this could mean splitting up long articles into paragraphs or sections, so each piece can be independently assessed for its relevance to a query.\nTransforming Text into Numeric Codes (Embedding Documents): We take each section of text and turn it into \u0026rsquo;embeddings\u0026rsquo; or numeric codes (vectors). This process is like giving each chunk its own unique number that reflects what the words are about.\u0026quot;\nStore the Embeddings: Save this embeddings into a special database called vector database like chromes[LINK]\nEmbedding the Query: Convert the query to embedding using the same method.\nFinding the Relevant Chunks: With both documents and the query turned into vectors, the retrieval system performs a search to identify which document embeddings are closest to the query embedding. Selection the information to use: Select top K items from the list that are most relevant to the query.\nAugmentation: # Append the information as context to the original prompt and pass it to the LLM.\nGeneration: # Generic LLM uses the prompt (original user query with relevant information) and generates the response.\nHope this article has helped you to understand RAG architecture. In the next article we will create a RAG application from scratch with no complex tools.\nStay tuned!!\nVideo Explaination # ","date":"3 June 2024","externalUrl":null,"permalink":"/post/rag-introduction/","section":"Post","summary":"RAG, or Retrieval Augmented Generation, is getting a lot of talk these days. It’s pretty exciting, but also a bit confusing for beginners. In this article I attempt to explain RAG in a way that’s easier to understand, without all the complex tech words.","title":"Retrieval Augmented Generation (RAG): A beginner’s guide to this complex architecture.","type":"post"},{"content":" Welcome to My Tech-Journal # A space where I share my thoughts, experience and learnings on technolgy and life.\n16 Projects built 89 Posts written 2 Years writing 2/6 Ambitions shipped ","date":"24 May 2024","externalUrl":null,"permalink":"/","section":"Home Page","summary":"","title":"Home Page","type":"page"},{"content":"","date":"24 May 2024","externalUrl":null,"permalink":"/tags/index/","section":"Tags","summary":"","title":"Index","type":"tags"},{"content":"","date":"24 May 2024","externalUrl":null,"permalink":"/post/","section":"Post","summary":"","title":"Post","type":"post"},{"content":" Introduction: # In our previous article, we explored how to create custom models using Ollama. If you haven’t read it yet, I encourage you to start there. Now, let’s expand on that knowledge by crafting a frontend chatbot application for our Drupal Code Assistant. Let\u0026rsquo;s dive right into coding it.\nRequirements: # We’ll be using Langchain and Streamlit for this tutorial.\nLangchain: Langchain is a framework designed to simplify the creation of applications using large language models. Learn more about Langchain.\nStreamlit: An open-source Python library that enables quick creation of web applications for data science and machine learning. Learn more about Streamlit\nSetting up the Environment: # First, we need to install the necessary packages:\npip install streamlit\npip install langchain_community\nBuilding the Chatbot: # Create a file chatbot.py and insert the following code. We\u0026rsquo;ll walk through each part step by step.\nimport streamlit from langchain_community.llms import ollama import time # App title streamlit.set_page_config(page_title=\u0026#34;Drupal Code Assistant\u0026#34;) with streamlit.sidebar:streamlit.title(\u0026#34;Drupal Code Assistant\u0026#34;) llm = ollama.Ollama(model=\u0026#34;drupal-code-assistant\u0026#34;) # Store llm generated messages. if \u0026#34;messages\u0026#34; not in streamlit.session_state.keys(): streamlit.session_state.messages = [{\u0026#34;role\u0026#34;: \u0026#34;assistant\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;How can i help you?\u0026#34;}] # Display the messages stored in streamlit messages session. for message in streamlit.session_state.messages: with streamlit.chat_message(message[\u0026#34;role\u0026#34;]): streamlit.write(message[\u0026#34;content\u0026#34;]) # Reset the messages session to null. def clear_chat_history(): streamlit.session_state.messages = [{\u0026#34;role\u0026#34;: \u0026#34;assistant\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;How can i help you?\u0026#34;}] # Add clear button. streamlit.sidebar.button(\u0026#34;Clear chat history\u0026#34;, on_click=clear_chat_history) def generate_llm_response(prompt): response = llm.invoke(input=prompt) return response # Check if the user has entered any input and if yes, assign it to prompt. if prompt := streamlit.chat_input(): streamlit.session_state.messages.append({\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: prompt}) with streamlit.chat_message(\u0026#34;user\u0026#34;): streamlit.write(prompt) if streamlit.session_state.messages[-1][\u0026#34;role\u0026#34;] != \u0026#34;assistant\u0026#34;: with streamlit.chat_message(\u0026#34;assistant\u0026#34;): response = generate_llm_response(prompt) # to show generating of response, char by char. placeholder = streamlit.empty() bot_response = \u0026#34;\u0026#34; for char in response: bot_response += char placeholder.markdown(bot_response) time.sleep(0.02) # to give writing feeling. Adjust as needed. message = {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: bot_response} streamlit.session_state.messages.append(message) To see the application in your browser, run:\nstreamlit run chatbot.py\nIf you encounter the \u0026ldquo;command not found: streamlit\u0026rdquo; error, use:\npython3 -m streamlit run chatbot.py\nCongratulations! Your chatbot is now ready for interaction.\nLet\u0026rsquo;s decode it word by word: # First, import the libraries.\nimport streamlit from langchain_community.llms import ollama import time This will set the page title and the side bar title of the chatbot.\nstreamlit.set_page_config(page_title=\u0026#34;Drupal Code Assistant\u0026#34;) with streamlit.sidebar:streamlit.title(\u0026#34;Drupal Code Assistant\u0026#34;) Create the ollama object and pass the name of our model drupal-code-assistant\nllm = ollama.Ollama(model=\u0026#34;drupal-code-assistant\u0026#34;) Now, we check if the messages key exists in the session state. If not, we initialise it with a greeting message from the assistant. In Streamlit, st.session_state is used to preserve state across reruns.\nif \u0026#34;messages\u0026#34; not in streamlit.session_state.keys(): streamlit.session_state.messages = [{\u0026#34;role\u0026#34;: \u0026#34;assistant\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;How can i help you?\u0026#34;}] Here, we render all the messages stored in streamlit.session_state.messages using a for loop. Each message contains two elements: \u0026lsquo;role\u0026rsquo; and \u0026lsquo;content.\u0026rsquo; The application features two roles: \u0026lsquo;user,\u0026rsquo; representing the individual interacting with the chatbot, and \u0026lsquo;assistant,\u0026rsquo; representing the chatbot itself.\nfor message in streamlit.session_state.messages: with streamlit.chat_message(message[\u0026#34;role\u0026#34;]): streamlit.write(message[\u0026#34;content\u0026#34;]) This will add a button labeled \u0026lsquo;Clear chat history,\u0026rsquo; and clicking on it will clear the chat and display the default message, \u0026lsquo;How can I help you?\u0026rsquo;\n# Reset the messages session to null. def clear_chat_history(): streamlit.session_state.messages = [{\u0026#34;role\u0026#34;: \u0026#34;assistant\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;How can i help you?\u0026#34;}] # Add clear button. streamlit.sidebar.button(\u0026#34;Clear chat history\u0026#34;, on_click=clear_chat_history) This function will be used later in the code to generate a response from the LLM model.\ndef generate_llm_response(prompt): response = llm.invoke(input=prompt) return response We use the walrus operator (:=) to check if there is any input in the chat (that is not None or an empty string). If input is present, it\u0026rsquo;s assigned to the \u0026lsquo;prompt\u0026rsquo; variable, and the program proceeds with the \u0026lsquo;if\u0026rsquo; condition. When the user submits some text, making \u0026lsquo;prompt\u0026rsquo; true, a new message is added to \u0026lsquo;st.session_state.messages\u0026rsquo; with the role of \u0026lsquo;user\u0026rsquo; and the content of the \u0026lsquo;prompt\u0026rsquo;.\nif prompt := streamlit.chat_input(): streamlit.session_state.messages.append({\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: prompt}) with streamlit.chat_message(\u0026#34;user\u0026#34;): streamlit.write(prompt) If you recall, we\u0026rsquo;ve already initialised \u0026lsquo;streamlit.session_state.messages\u0026rsquo;. Now, we check if the role of the last message is \u0026lsquo;user\u0026rsquo; (and not \u0026lsquo;assistant\u0026rsquo;). If that\u0026rsquo;s the case, we continue by displaying a placeholder for the assistant\u0026rsquo;s response with \u0026lsquo;streamlit.chat_message(\u0026ldquo;assistant\u0026rdquo;).\u0026rsquo; We then fetch the response from \u0026lsquo;generate_llm_response()\u0026rsquo; and simulate the chatbot typing out the message by gradually filling the placeholder, character by character, with a time delay of 0.02 seconds between each. Feel free to adjust the speed to your preference.\nif streamlit.session_state.messages[-1][\u0026#34;role\u0026#34;] != \u0026#34;assistant\u0026#34;: with streamlit.chat_message(\u0026#34;assistant\u0026#34;): response = generate_llm_response(prompt) placeholder = streamlit.empty() bot_response = \u0026#34;\u0026#34; for char in response: bot_response += char placeholder.markdown(bot_response) time.sleep(0.02) # to give writing feeling. Adjust as needed. message = {\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: bot_response} streamlit.session_state.messages.append(message) And that\u0026rsquo;s it! We\u0026rsquo;ve successfully created a chatbot that runs locally on our own computer.\nFinal Thoughts: Wrapping up this three-part series, we\u0026rsquo;ve learned what Ollama is, how to create local LLMs, and how to design chatbot applications like ChatGPT with it—all locally and free!\nVideo Explaination # If you want video version of this content, check out this video.\n","date":"24 May 2024","externalUrl":null,"permalink":"/post/ollama-chatbot/","section":"Post","summary":"In our previous article, we explored how to create custom models using Ollama. If you haven’t read it yet, I encourage you to start there. Now, let’s expand on that knowledge by crafting a frontend chatbot application for our Drupal Code Assistant. Let’s dive right into coding it.","title":"Drupal Code Assistant: Build Chatbot using Ollama","type":"post"},{"content":"","date":"24 May 2024","externalUrl":null,"permalink":"/tags/drupal10/","section":"Tags","summary":"","title":"Drupal10","type":"tags"},{"content":"","date":"24 May 2024","externalUrl":null,"permalink":"/series/ollama/","section":"Series","summary":"","title":"Ollama","type":"series"},{"content":"Hi, I am Akansha.\nI am a cat-mom to Green and Chiggu I am a software developer. I am a runner who also likes swimming and strength training. Meet the cats # 🐈 Green Senior code reviewer. Naps through every standup. Approves PRs by walking across the keyboard. 🐈‍⬛ Chiggu Head of security. Guards the yarn drawer. Treats every closed door as a personal insult. When not working you will find me cribbing about my ambitious nature. I have many ambitions, some of those make it to real life, some remain in the list. Below are the ones that come to my mind while typing:\nDo Everest Base Camp trek. Feel satisfied with the work at my job. Build Find-My-Furr, a tool to help pet-parents help find their lost pet. RunStrengthLab - An app to help runners do strength training. Greencard - Personal Jeera Become a better Engineer. To feel good, I build software. Those may or may not solve a real problem, but definitely help with my imposter syndrome. You can find those at Side Projects\n","date":"24 May 2024","externalUrl":null,"permalink":"/about/","section":"Home Page","summary":"","title":"About","type":"page"},{"content":" Introduction: # In the previous article, we learned how easy it is to use Ollama for running large language models on our own computers. This time around, we\u0026rsquo;re taking things a step further by creating a custom LLM model tailored just for us. Let\u0026rsquo;s dive into this hands-on guide!\nUse Case: # As a Drupal Developer, my day includes coding, reviewing, and debugging - all in a day\u0026rsquo;s work. But what if I had a reliable assistant to help me out? Someone who could review my code, find bugs, suggest improvements, or even generate new code!\nLet’s see how Ollama can make the job easy.\nGetting Started with Your Custom Model: # First things first, find a model that\u0026rsquo;s close to what you need in the Ollama Library. We\u0026rsquo;ll use \u0026lsquo;codellama\u0026rsquo;, a model by Meta, openly available LLM to generate and discuss code.\nHere\u0026rsquo;s how we roll with Ollama:\nChoosing a Model: Look for \u0026lsquo;codellama\u0026rsquo; in the Ollama Library. It\u0026rsquo;s going to be our base model.\nSelecting Parameters: By default the Ollama pulls the latest tag but if needed, you can select a specific one.\nPull the latest tag: ollama pull codellama\nPull with specific tag: ollama pull \u0026lt;model\u0026gt;:\u0026lt;tag\u0026gt;\nBuilding an Assistant: # We\u0026rsquo;re building a custom model DrupalCode Assistant that understands Drupal development to the core. Follow the steps:\nCheck out the Model File for codellama using: ollama show --modelfile codellama Create your Modelfile and set the parameters like this: # Modelfile for creating a Drupal Code Assistant FROM codellama PARAMETER temperature 0.7 SYSTEM You are a Drupal developer expert, acting as an assistant. You offer help with generating code, reviewing code and providing suggestions, generating unit test cases and explaining code. You answer with code examples when possible. FROM (Required): defines the base model name with an optional \u0026rsquo;tag\u0026rsquo; to run. Syntax: FROM \u0026lt;model name\u0026gt;:\u0026lt;tag\u0026gt; PARAMETER: the parameter instruction defines the parameters that can be set when the model is run. Syntax: PARAMETER \u0026lt;parameter\u0026gt; \u0026lt;parametervalue\u0026gt; We are using temperature to set the value of the creativity. Increasing the temperature will make the model answer more creatively. (Default: 0.7). Access list of all the valid parameters and values from here. SYSTEM: The SYSTEM instruction sets the custom system message to specify the behaviour of the assistant. We want to our assistant to revolve around Drupal. Deploy the model: # Using command ollama create drupal-code-assistant -f ./Modelfile\nNow that your model has been created, you can check it out by listing all available models with ollama list.\nRun the Model: # Get it running using command ollama run drupal-code-assistant\nTesting Out Your Model: # Here\u0026rsquo;s how you can start conversations with your Drupal Code Assistant:\nExample 1: Requesting Code\nPrompt: Can you help me write a Drupal Controller to display articles tagged with term ID 123 in table format? Response:\nExample 2: Review code.\nPrompt: Can you review this code public static function createInstance(ContainerInterface $container, EntityTypeInterface $entity_type) { return new static( $entity_type, $container-\u0026gt;get(\u0026#39;entity_type.manager\u0026#39;)-\u0026gt;getStorage($entity_type-\u0026gt;id()), $container-\u0026gt;get(\u0026#39;plugin.manager.action\u0026#39;) ); } Response:\nYour assistant should now provide responses tailored to the Drupal development context.\nFinal Thoughts: Remember where we started with simple tasks on Ollama? Now you\u0026rsquo;ve got a full-on Drupal-savvy assistant right in your local environment. Stay with me, and in the next article, we will create the front-end of this assistant. Until then, enjoy your new assistant!\nVideo Explaination # If you want video version of this content, check out this video.\nResources Used: # https://github.com/ollama/ollama/blob/main/docs/modelfile.md https://unmesh.dev/post/ollama_custom_model/ https://www.gpu-mart.com/blog/custom-llm-models-with-ollama-modelfile https://copilot.microsoft.com/ ","date":"14 May 2024","externalUrl":null,"permalink":"/post/ollama-custom-model/","section":"Post","summary":"As a Drupal Developer, my day includes coding, reviewing, and debugging - all in a day’s work. But what if I had a reliable assistant to help me out? Someone who could review my code, find bugs, suggest improvements, or even generate new code!\nLet’s see how Ollama can make the job easy.","title":"AI Helper: Build your own Custom LLM Model on Ollama","type":"post"},{"content":" Use Cases: # Imagine it’s early in the morning, and you\u0026rsquo;re thirsty for the latest news on Drupal. You\u0026rsquo;d usually hop onto Drupal.org or sift through various websites to find quality articles. But you only want the freshest, top-notch reads from this week. ChatGpt, Microsoft Copilot or any other big tech companies\u0026rsquo; LLM tool can help with that. But here’s the twist - with Ollama, you can get that, for free and even automate to email you those articles every-day.\nNow, let\u0026rsquo;s say you\u0026rsquo;re knee-deep in coding for a Drupal project. Occasionally, you hit a roadblock or need to decipher someone else\u0026rsquo;s code. Instead of scrolling through endless forums or staring at your screen puzzled, Ollama is your go-to buddy. Just like how you\u0026rsquo;d ask ChatGPT for help, Ollama can review your code or clarify tricky parts, all locally on your machine.\nOr, imagine you\u0026rsquo;re writing code for a Drupal site and need a second pair of eyes to check your work. Maybe you’re stuck or need to understand what someone else’s code does. Instead of search the web or endless threads on stack overflow. With ollama you can get your answers quickly and locally.\nOllama is open-source, which means it’s there for everyone to use, whether it’s for personal projects or bigger work tasks.\nGet Ollama on Your Computer # Visit Ollama download page. Select the download that fits your operating system. Download the file, open it, and follow the installation steps. Running Ollama: # To get Ollama working, do this:\nOpen your computer\u0026rsquo;s terminal window. Type in Ollama run llama3 and press Enter. Your computer will get set up with LLAMA3. Try different models: # Check out the Ollama Library to see what\u0026rsquo;s out there. Choose a model and get it with a command like ollama pull phi3. This will download the manifest of the model at your local. Now Run it with Ollama run \u0026lt;model_name\u0026gt;:\u0026lt;tag\u0026gt;. That\u0026rsquo;s all for now. In our next article, we\u0026rsquo;ll explore how to create custom LLM models using Ollama\nVideo Explaination # If you want video version of this content, check out this video.\n","date":"4 May 2024","externalUrl":null,"permalink":"/post/ollama-introduction/","section":"Post","summary":"Imagine it’s early in the morning, and you’re thirsty for the latest news on Drupal. You’d usually hop onto Drupal.org or sift through various websites to find quality articles. But you only want the freshest, top-notch reads from this week. ChatGpt, Microsoft Copilot or any other big tech companies’ LLM tool can help with that. But here’s the twist - with Ollama, you can get that, for free and even automate to email you those articles every-day.","title":"Introducing Ollama: The Local Powerhouse for Language Models","type":"post"},{"content":"","externalUrl":null,"permalink":"/english/","section":"","summary":"","title":"","type":"english"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"}]