Data is the fundamental building block of any AI system. The quality, quantity, and relevance of the data directly impact the performance and reliability of AI models. The data collection and acquisition phase is therefore paramount, focusing on identifying, sourcing, and gathering the necessary data to train, validate, and test AI models effectively. This phase is not merely about accumulating vast amounts of data; it’s about acquiring the right data that accurately represents the problem space and is suitable for the intended AI application.
Identifying and Sourcing Data #
The process of data collection begins with a thorough understanding of the data requirements derived from the problem definition phase. This involves identifying what types of data are needed, what formats they should be in, and what volume is required to achieve the desired model performance. Data can originate from a multitude of sources, both internal and external to an organization.
Internal Data Sources #
- Operational Databases: Transactional data, customer records, sales figures, and operational logs from enterprise resource planning (ERP), customer relationship management (CRM), and other internal systems.
- Data Warehouses/Lakes: Centralized repositories designed for analytical processing, often containing historical and aggregated data.
- Application Logs: Data generated by software applications, including user interactions, system events, and error logs.
- IoT Devices: Sensor data from connected devices in manufacturing, smart cities, healthcare, and other domains.
- Customer Interactions: Call center recordings, chat transcripts, email communications, and social media interactions (if managed internally).
External Data Sources #
Keep reading with a free Eristotle account.
Every document across our Bodies of Knowledge is free to read once you are signed in. No payment, no tier.
