In the digitalization journey of the world, the data size journey that we started with bits continues to be named with data size concepts such as gigabyte, terabyte and even petabyte, exabyte, zettabyte, yottabyte, which we have never heard of. We will probably hear different concepts for larger ones in the coming years. Of course, in addition to the increase in these sizes, our data sources are diversifying day by day. From the smartwatch on our wrists to the security camera of a shopping center we enter, a lot of data is collected from many different environments that we are aware of but not aware of. And this chain will continue to grow within the Internet of Things… Of course, the data we systematically collect from our customers or from some open sources on the internet is also a link in this chain.
On the other hand, companies have also benefited from this digitalization, and there have been companies that have made serious breakthroughs in decision-making in this data pool that reaches them through both the data they keep about their customers and digital objects, and the number of companies that understand the power of this data is increasing day by day.
Of course, in parallel with the increase in our data sources, some problems started to emerge and the importance of collecting data with an effective strategy started to increase as ecosystems grew. Of course, this situation has brought different questions to our minds and turned into an obligation to move forward with new approaches at this point. While there are many questions on this topic, a few more issues are often at the forefront of decision-makers’ minds at the point of data collection.
First of all, yes, we are collecting this much data, but which data is essential for our project and how can we store really usable data in the right way and save our systems from unnecessary burden while saving us from increasing storage costs and converting this data into money for our company in line with our strategies?
This is where data retention strategies for companies come into play. Although sectors and companies have their own different dynamics, there are some issues that every company should pay attention to when determining the strategy on this issue. Let’s look at these now. Of course, the order of the items I have mentioned here may change according to the project and the company, but in general terms, it is possible to address these issues under the following headings.
Necessity
Security
Standardization
Sustainability
Quality
Usability and Integration
NECESSITY
The first question is, do we really need this data for our projects?
As I mentioned at the beginning of the article, the resources we can now collect data are moving towards the point of being unlimited. However, just as our storage areas cannot grow at the same percentage, the system resources we develop do not grow at the same rate at the point of processing data, or companies may delay investing in systems at the point of profitability. At this point, there are some decisions that need to be taken.
First of all, project needs need to be clearly defined. The variety of data required for a data science project may not require the same depth of data for a mobile application project to be developed from b2b. Therefore, for each project, companies need to think “Yes. This is the data I need” rather than “Will I need this data one day? This is the data I need. And I can keep this data within my project budget.” It is important that they adopt the understanding. Of course, since projects may lead to other projects and needs over time, it is equally necessary to establish flexible and scalable structures. Project managers and business analysts have important roles to play here. In summary, a good analysis of the needs is the most necessary condition.
In the opposite scenario, it is very likely that your company will face a lot of idle data that no one knows why it is there, as well as idle capacity, labor and cost issues.
SAFETY
At the next stage, we store this data, but it is important to look for answers to the questions of whether our data comes from secure sources and can we ensure the security of the data we hold?
Our data source is very small, our data capacity and system is small. Okay, we have clearly determined the data we need, and we have created a flexible and scalable architecture considering the projects that may arise in the future. So at what level can we ensure the security of the data we keep? And do we store the data where it should be?
Data security has become an important issue for the whole world in recent years and companies are sometimes subject to serious sanctions for failing to protect some of the data they provide from their users. The dose of cyber-attacks is increasing day by day and the issue of data security is becoming even more important. On the other hand, companies working with different countries in terms of the security of the data kept here need to simultaneously follow not only the data security protocols in their own countries but also the protocols of the countries where their users are located. And when necessary, they may have to keep their data in other countries. Although cloud solutions and data centers make this relatively easier, they may not always offer a 100% solution. Since this part now falls under data law, legal support is very important as well as database administrators.
Two questions arise here. How secure are the data sources we will use and how secure are the places where we will save this data?
Yes, we want to transfer the data we need from a source that we think is reliable to the database we use and shape our project with such a structure. Well, have we ever thought about the possibility that the data source used could create a vulnerability in our system or generate incorrect data?
This is a serious problem and one that really needs to be thought about. If it is not secure, that data is never yours. This is where the importance of cyber security experts comes into play. On the other hand, database administrators and data engineers also have a lot of work to do in securely retrieving the data from the source and protecting it from attacks. As I said, no unsecured data belongs to you. And data that is thought to be cheap today may cost you serious costs in the future.
STANDARDIZATION
We have identified the data required for our project and securely stored it. So, does our data have a standardization to talk to the modules within our project and the customer?
This is one of the problems faced by companies working with data in recent years. We talked about the abundance of data sources. But every data source you consult outside the company may have a data architecture created as a result of its own dynamics. So, can we use the data we get from so many sources as it is? I seem to hear you wish, but this is not possible. In fact, it is almost never possible.
This is where data engineering and data architecture come into play. Even if the project manager and business analysts determine the needs in the structure to be established, it is very valuable to make data from different sources available to other developers involved in the project with the least effort. Otherwise, it should not be forgotten that your development team will exert more effort to work with this data, which may lead the project to dead ends in terms of both time and budget. In summary, the data kept should speak the same language as much as possible. And all developers working with the same data should be able to understand the same things from that data.
SUSTAINABILITY
The next issue in data retention strategies is sustainability. This is the unknown gateway to unlimited data sources. But there are always safe harbors.
Of course, there are two dimensions to the issue.
First of all, while the dynamic data we keep will grow day by day, will we be able to keep up with this growth? We started the project with a 10 gigabyte space, but what is our plan B when our disk starts to fill up before a few months have passed? Will we move this data to another disk? Or will we enlarge this space by paying a little more for the data we keep on the cloud? Or will we produce another solution? It should not be forgotten that all 3 options should be considered according to the project. The decisions to be made here are facts that decision makers should definitely not ignore in terms of time, cost and effort.
But what about the reliability of our data source? I have already touched on the cyber-attack and legal aspects of security, but here we also need to ask the following questions. How long will I be able to access the data source we use? In other words, tomorrow the data sharing policies of the source from which you get your data may change and you may have to pay a fee to get data from that source, or you may not get any data at all and you may have to turn to other sources. At this point, choosing a well-established and well-established data source can save you a lot of potential costs in the future. Data analysts also have a role to play here.
QUALITY
Under the heading of standardization, I mentioned the importance of data sources where data can be transformed into a form that can be used by all stakeholders with the least effort. So how does data quality appear here?
There is a situation that is important. There may be incorrect data in the data we receive from different sources or there may be some inconsistencies in the templates of the data. Yes, the source from which you get the data is the same, but no correction has been applied for the data coming from the source, or this has been done partially, and you need extra effort to constantly check and work on the data, apart from your standardization stages.
To give a simple example; let’s say you wanted to use air temperatures from a source in your data project. Yes, the data flows every day to the source from which you get the data, and you directly transfer this data to your own database and advance your project from there. You never questioned the data quality of the source and you wanted to use it as it is…
This is where you may need to be ready for some surprises.
There may be missing numerical data, incorrect entries, or one part of the data may be entered in one numerical type while another part is entered in another data type. Of course, until you realize these and integrate them into your own system, the time that will pass in between, or even the incorrect design of your system in places, may require you to bear some additional costs. Remember, sometimes even the spaces next to the row values in a simple excel table can be overlooked and we have problems until we find them. Or working with clean data by constantly extracting outlier values can be a waste of time.
COMPATIBILITY AND INTEGRATION
Finally, the compatibility of the data we will use with the technologies we use is also an important parameter in terms of data storage strategies. Yes, data may be available in an external source and it may be there under the conditions we want. However, some of the software solutions required to process this data are not available in our company, or even if they are available, the potential time and labor costs that may arise at the point of adaptation to our system are among the points to be considered.
At this point, the use of resources that can be more easily integrated with the technologies used by your company may be an important reason for preference. Here, it is important to make effective cost-oriented and strategic decisions from time to time for the salvation of the project. Because changing the technology used in the middle of projects is not a situation that developers would prefer. It is also possible that adapting to the new technology may cause some potential additional costs in terms of time, effort and budget.
In addition, the ability of the staff to quickly adapt to the existing ecosystem after the departure of the developer working on the project is also an issue that should be considered under this heading.
At the end of the day, there is always the possibility that the data we cannot collect with certain strategies will lead us to unknowns in terms of budget, time and effort for our project, although the order of these headings will vary according to the companies.
For this reason, while creating our projects, regardless of the scale of the project, whether we are a freelancer or the IT department manager of an international company, the above-mentioned issues will be the issues that we all need to question and act with strategic planning in terms of proper management of both labor, system resources and budget.
DATA RETENTION STRATEGIES
In the digitalization journey of the world, the data size journey that we started with bits continues to be named with data size concepts such as gigabyte, terabyte and even petabyte, exabyte, zettabyte, yottabyte, which we have never heard of. We will probably hear different concepts for larger ones in the coming years. Of course, in addition to the increase in these sizes, our data sources are diversifying day by day. From the smartwatch on our wrists to the security camera of a shopping center we enter, a lot of data is collected from many different environments that we are aware of but not aware of. And this chain will continue to grow within the Internet of Things… Of course, the data we systematically collect from our customers or from some open sources on the internet is also a link in this chain.
On the other hand, companies have also benefited from this digitalization, and there have been companies that have made serious breakthroughs in decision-making in this data pool that reaches them through both the data they keep about their customers and digital objects, and the number of companies that understand the power of this data is increasing day by day.
Of course, in parallel with the increase in our data sources, some problems started to emerge and the importance of collecting data with an effective strategy started to increase as ecosystems grew. Of course, this situation has brought different questions to our minds and turned into an obligation to move forward with new approaches at this point. While there are many questions on this topic, a few more issues are often at the forefront of decision-makers’ minds at the point of data collection.
First of all, yes, we are collecting this much data, but which data is essential for our project and how can we store really usable data in the right way and save our systems from unnecessary burden while saving us from increasing storage costs and converting this data into money for our company in line with our strategies?
This is where data retention strategies for companies come into play. Although sectors and companies have their own different dynamics, there are some issues that every company should pay attention to when determining the strategy on this issue. Let’s look at these now. Of course, the order of the items I have mentioned here may change according to the project and the company, but in general terms, it is possible to address these issues under the following headings.
NECESSITY
The first question is, do we really need this data for our projects?
As I mentioned at the beginning of the article, the resources we can now collect data are moving towards the point of being unlimited. However, just as our storage areas cannot grow at the same percentage, the system resources we develop do not grow at the same rate at the point of processing data, or companies may delay investing in systems at the point of profitability. At this point, there are some decisions that need to be taken.
First of all, project needs need to be clearly defined. The variety of data required for a data science project may not require the same depth of data for a mobile application project to be developed from b2b. Therefore, for each project, companies need to think “Yes. This is the data I need” rather than “Will I need this data one day? This is the data I need. And I can keep this data within my project budget.” It is important that they adopt the understanding. Of course, since projects may lead to other projects and needs over time, it is equally necessary to establish flexible and scalable structures. Project managers and business analysts have important roles to play here. In summary, a good analysis of the needs is the most necessary condition.
In the opposite scenario, it is very likely that your company will face a lot of idle data that no one knows why it is there, as well as idle capacity, labor and cost issues.
SAFETY
At the next stage, we store this data, but it is important to look for answers to the questions of whether our data comes from secure sources and can we ensure the security of the data we hold?
Our data source is very small, our data capacity and system is small. Okay, we have clearly determined the data we need, and we have created a flexible and scalable architecture considering the projects that may arise in the future. So at what level can we ensure the security of the data we keep? And do we store the data where it should be?
Data security has become an important issue for the whole world in recent years and companies are sometimes subject to serious sanctions for failing to protect some of the data they provide from their users. The dose of cyber-attacks is increasing day by day and the issue of data security is becoming even more important. On the other hand, companies working with different countries in terms of the security of the data kept here need to simultaneously follow not only the data security protocols in their own countries but also the protocols of the countries where their users are located. And when necessary, they may have to keep their data in other countries. Although cloud solutions and data centers make this relatively easier, they may not always offer a 100% solution. Since this part now falls under data law, legal support is very important as well as database administrators.
Two questions arise here. How secure are the data sources we will use and how secure are the places where we will save this data?
Yes, we want to transfer the data we need from a source that we think is reliable to the database we use and shape our project with such a structure. Well, have we ever thought about the possibility that the data source used could create a vulnerability in our system or generate incorrect data?
This is a serious problem and one that really needs to be thought about. If it is not secure, that data is never yours. This is where the importance of cyber security experts comes into play. On the other hand, database administrators and data engineers also have a lot of work to do in securely retrieving the data from the source and protecting it from attacks. As I said, no unsecured data belongs to you. And data that is thought to be cheap today may cost you serious costs in the future.
STANDARDIZATION
We have identified the data required for our project and securely stored it. So, does our data have a standardization to talk to the modules within our project and the customer?
This is one of the problems faced by companies working with data in recent years. We talked about the abundance of data sources. But every data source you consult outside the company may have a data architecture created as a result of its own dynamics. So, can we use the data we get from so many sources as it is? I seem to hear you wish, but this is not possible. In fact, it is almost never possible.
This is where data engineering and data architecture come into play. Even if the project manager and business analysts determine the needs in the structure to be established, it is very valuable to make data from different sources available to other developers involved in the project with the least effort. Otherwise, it should not be forgotten that your development team will exert more effort to work with this data, which may lead the project to dead ends in terms of both time and budget. In summary, the data kept should speak the same language as much as possible. And all developers working with the same data should be able to understand the same things from that data.
SUSTAINABILITY
The next issue in data retention strategies is sustainability. This is the unknown gateway to unlimited data sources. But there are always safe harbors.
Of course, there are two dimensions to the issue.
First of all, while the dynamic data we keep will grow day by day, will we be able to keep up with this growth? We started the project with a 10 gigabyte space, but what is our plan B when our disk starts to fill up before a few months have passed? Will we move this data to another disk? Or will we enlarge this space by paying a little more for the data we keep on the cloud? Or will we produce another solution? It should not be forgotten that all 3 options should be considered according to the project. The decisions to be made here are facts that decision makers should definitely not ignore in terms of time, cost and effort.
But what about the reliability of our data source? I have already touched on the cyber-attack and legal aspects of security, but here we also need to ask the following questions. How long will I be able to access the data source we use? In other words, tomorrow the data sharing policies of the source from which you get your data may change and you may have to pay a fee to get data from that source, or you may not get any data at all and you may have to turn to other sources. At this point, choosing a well-established and well-established data source can save you a lot of potential costs in the future. Data analysts also have a role to play here.
QUALITY
Under the heading of standardization, I mentioned the importance of data sources where data can be transformed into a form that can be used by all stakeholders with the least effort. So how does data quality appear here?
There is a situation that is important. There may be incorrect data in the data we receive from different sources or there may be some inconsistencies in the templates of the data. Yes, the source from which you get the data is the same, but no correction has been applied for the data coming from the source, or this has been done partially, and you need extra effort to constantly check and work on the data, apart from your standardization stages.
To give a simple example; let’s say you wanted to use air temperatures from a source in your data project. Yes, the data flows every day to the source from which you get the data, and you directly transfer this data to your own database and advance your project from there. You never questioned the data quality of the source and you wanted to use it as it is…
This is where you may need to be ready for some surprises.
There may be missing numerical data, incorrect entries, or one part of the data may be entered in one numerical type while another part is entered in another data type. Of course, until you realize these and integrate them into your own system, the time that will pass in between, or even the incorrect design of your system in places, may require you to bear some additional costs. Remember, sometimes even the spaces next to the row values in a simple excel table can be overlooked and we have problems until we find them. Or working with clean data by constantly extracting outlier values can be a waste of time.
COMPATIBILITY AND INTEGRATION
Finally, the compatibility of the data we will use with the technologies we use is also an important parameter in terms of data storage strategies. Yes, data may be available in an external source and it may be there under the conditions we want. However, some of the software solutions required to process this data are not available in our company, or even if they are available, the potential time and labor costs that may arise at the point of adaptation to our system are among the points to be considered.
At this point, the use of resources that can be more easily integrated with the technologies used by your company may be an important reason for preference. Here, it is important to make effective cost-oriented and strategic decisions from time to time for the salvation of the project. Because changing the technology used in the middle of projects is not a situation that developers would prefer. It is also possible that adapting to the new technology may cause some potential additional costs in terms of time, effort and budget.
In addition, the ability of the staff to quickly adapt to the existing ecosystem after the departure of the developer working on the project is also an issue that should be considered under this heading.
At the end of the day, there is always the possibility that the data we cannot collect with certain strategies will lead us to unknowns in terms of budget, time and effort for our project, although the order of these headings will vary according to the companies.
For this reason, while creating our projects, regardless of the scale of the project, whether we are a freelancer or the IT department manager of an international company, the above-mentioned issues will be the issues that we all need to question and act with strategic planning in terms of proper management of both labor, system resources and budget.
Arşivler
Kategoriler
Arşivler
AUTONOMOUS CRUISE CONTROL IN AUTOMOBILES USING FUZZY LOGIC
22 Haziran 2025DATA RETENTION STRATEGIES
22 Haziran 2025Categories
Meta