Inside the Ivory Tower

George Floyd

2020-06-03T11:21:00-04:00

Dear College of Engineering students, staff, and faculty,

Watching the nation be torn apart is heartbreaking. I have spoken to my friends and advisors asking what I should do- what I should communicate. There is no training for an engineering dean to prepare for this kind of communication. There are the boilerplate phrases one should say- but they can’t reflect how I feel. Further, I simply do not have the eloquence of an English major, nor the diplomacy of a political science major. So, I will open my heart and share what I have to say, as rough as they may be.

The tragic loss of George Floyd on May 25^th was terrible, painful, and should have been a wholly unnecessary reminder that we as a nation have not yet become a society without discrimination. Our nation has been moving towards this goal for many centuries, and there should be no doubt it has taken far too long. There should be no doubt that we cannot arrive there fast enough. While all of us face challenges in our lives, some far greater than others, many of us live our lives oblivious to the undue burdens placed on others due to their race, gender, religion, and sexual orientation. I say us because while I don’t suffer this way, I refuse to define those who do as “them”. Us, and them. It is a way of dividing us and enabling the worst of our behaviors to find a toehold in our hearts. There cannot be a “them” in this.

I have dear friends from a variety of races; multiple friends who are gay; and many who are Muslim, Atheist, Buddhist, and Hindu. Many do not share my gender. I am also friends with multiple police officers. Each of my friends has touched me in ways over my life to make me a much better person- one who continues to need to improve.

Those who I have been able to reach out to or hear from are appalled and heartbroken by the treatment of Mr. Floyd and what we all see and read. My heart is broken because I see so much anger rising up where is has no place. There is room here for anger, no doubt. But there is no room for the hatred that is being demonstrated, and much of the anger is misguided and mistargeted. We see it applied against police as a whole by some, and all protesters by others. We see vandals and rioters being confused with the peaceful protestors of #BlackLivesMatter. We see journalists covering these events also being assaulted and wrongly arrested. We see members of the police, the vast majority of whom are good and decent people who would not perpetrate such an act. Please do not fall into the tendency to group individuals or assess members of a group by the actions of a few. That is one of the pillars of discrimination.

This anger and hatred is heartbreaking to see. Hatred is one of the ugliest of feelings, and one of the most damaging to one’s self. I am grateful, though, to also have seen much of the best of humanity during these painful events. Trying to make sense of this I have watch numerous videos and read numerous articles. I have seen peaceful protestors who stopped a vandal during a protest arrest the criminal and turn them over to the police. I watched the police chief of Flint, MI set down his helmet and gear and join the protestors in solidarity. I listened to a passionate young woman in Louisville talk to soldiers and watched as they respond by taking a respectful knee with her in solidarity. Watching miscreants attempt to hijack peaceful and righteous protests to create anarchy, stir up hatred, and exercise power for its own sake is terrible. While there are surely many pretending to be a part of group, pretending to espouse their beliefs and vows, they are still few compared to the rest of us. We cannot allow them to drive our dialog. We cannot allow them to penetrate our hearts.

Those who are discriminated against know full well that recent events have just shined the light on the darkness that walks amongst us- they are far more likely to see that evil and be victims of it than the rest of us. As hard as these events are, there is some good having this light shone brightly. It raises awareness so that it cannot be ignored. It behooves all of us to look for the solutions to the problems of the aggrieved instead of blaming them.

There is only one path forward, and that is love. That is the message of Jesus Christ, Mahatma Gandhi, and Martin Luther King. If we truly love one another, with full sincerity, we will come through this better in the end. True love means patience. It means not rushing to judgement and respecting individuals for themselves. It means we truly listen to the expressions of anguish and pain that we hear- and that we ask questions conveying that we truly care about one another. It is especially important to try to understand those who do not agree with us. Only through growing mutual understanding can we respectfully work towards a better future.

I cannot promise an outcome. I can promise that if we embrace inclusiveness, we will be a better people for it, and a better college. If we embrace true love for one another, we will be even better yet. These are my hopes for everyone during this time, but especially so for the college. We will never be stronger than our ability to embrace those who are different from us. I read in one tweet this morning a quote from Benjamin Franklin that resonates as much today as it did over 200 years ago: “Justice will not be served until those who are unaffected are as outraged as those who are.” I would add to that a lesson I have learned over my life: the greatest reward we receive is by giving to others. Right now, those of us who are not amongst those who feel targeted or afraid must give. We must lend an ear, and give our hearts. There is no other good path forward.

Sincerely,

Joseph C. Slater

Dean- College of Engineering

Tennessee Technological University

Curve fitting experimental data

2018-09-07T15:00:00-04:00

Curve fitting data is a standard practice in creating physical models of observed phenomena. When an expected model form exists, we use that form and fit it the best we can. If there is too much error for our needs, we may ignore the physical model for the sake of expedience. If we have no physics-based model we are free to use what ever seems to work the best. Often the solution presented is to plot data, pick two points, then get a line from that. Picking two points “works”, but it’s crass and defeats the point of collecting that much data. A much better way is to transform the data using log functions, find your parameters, and transform them appropriately from log space.

Fitting a line

Let’s start with a refresher for curve fitting \(y=mx+b\). Our derivation led us to two equations and two unknowns. You may use those. With a modern computer (under 35 years old), the faster way is to do this with linear algebra. With Matlab, polyfit will do this for you. I will use this later.

With linear algebra, we write the equations such that:

\begin{equation*} y_i = m x_i + b \end{equation*}

Putting a bunch of these together, I can write them in matrix form as

\begin{equation*} \begin{bmatrix} y_1\\ y_2\\ y_3\\ y_4\\ \vdots \end{bmatrix} = \begin{bmatrix} x_1& 1\\ x_2& 1\\ x_3& 1\\ x_4& 1\\ \vdots&\vdots\\ \end{bmatrix} \begin{bmatrix} m\\ b\end{bmatrix} \end{equation*}

\begin{equation*} which for compactness we write as \end{equation*}

\begin{equation*} Y = X B \end{equation*}

Because \(X\) is not square, we can instead pre-multiply by \(X^T\) so that the matrix preceding \(B\) is now square (specifically, for a line it is now a \(2\times 2\) matrix).

\begin{equation*} X^TY=X^TXB \end{equation*}

Pre-multiplying both sides by \((X^TX)^{-1}\) we have

\begin{equation*} (X^TX)^{-1}X^TY=B \end{equation*}

The elements of \(B\) are \(m\) and \(b\). In Matlab, this would look something like

>> Y = [1;2;3;4]
Y =
   1
   2
   3
   4
>> x = [4;3;2;1]
x =
   4
   3
   2
   1
>> X = [x*0+1, x]
X =
   1   4
   1   3
   1   2
   1   1
>> B = (X'*X)\X'*Y
B =
   5
  -1

which means a \(y\) intercept of \(5\) an a slope of \(1\).

We could alternatively have used polyfit like this:

>> B = polyfit(x, Y,1)
B =
  -1.0000   5.0000

While this is certainly simpler, and allowed for the course it is limited in its flexibility compared to the detailed method which would allow you to fit a 2-D surface instead of just a line.

To curve fit a parabola, you follow the same procedure as above, but starting with \(y = a x^2 + b x + c\), or you use polyfit with the last parameter set to \(2\).

Other forms

Power function

I covered how to do this in class when you want a model of the form \(y=bx^m\).

Let’s refresh:

Start by taking the natural log of the equation. This yields

\begin{equation*} \ln(y)=\ln(b)+m \ln(x) \end{equation*}

For simplicity, you could define \(\ln(y)=Y\), \(\ln(x)=X\), and \(\ln(b)=B\) so that the equation is

\begin{equation*} Y = m X + B \end{equation*}

The is the equation of a line. If you use the log of the data \(x\) and \(y\), and curve fit like you would a line, you obtain \(m\) and \(\ln(b)\). Since \(b=e^{B}=e^{\ln(b)}\), if you can fit a line, you can get the parameters \(b\) and \(m\) for the model \(y=bx^m\).

Exponential function

Let’s start by presuming a model of the form

\begin{equation*} y = b e^{mx} \end{equation*}

Start by taking the natural log of the equation. This yields

\begin{equation*} \ln(y)=\ln(b)+m x \end{equation*}

For simplicity, you could define \(\ln(y)=Y\) and \(\ln(b)=B\) so that the equation is

\begin{equation*} Y = m x + B \end{equation*}

The is the equation of a line. If you use the \(\ln\) of the data \(y\), and curve fit like you would a line, you obtain \(m\) and \(\ln(b)\). Since \(b=e^{B}=e^{\ln(b)}\), if you can fit a line, you can get the parameters \(b\) and \(m\) for the model \(y=be^{xm}\).

In Matlab, this would look like

x = (0:10)'
y =[3.0138902;
   1.8036854;
   1.0875830;
   0.6749870;
   0.4167889;
   0.2215530;
   0.1667404;
   0.0906666;
   0.0593329;
   0.0275748;
   0.0046658]
>> B = polyfit(x, Y, 1)
B =
  -0.57030   1.30850
>>> b = exp(B(2))
b =  3.7006

Here, \(1.30850\) is not the \(y\) intercept but instead is the \(\ln\) of the coefficient to the exponential (still called \(b\)). So the equation is

\begin{equation*} y(x) = 3.7006 e^{(-0.57030 x)} \end{equation*}

Most students in engineering schools seem to be using Matlab, so I have presented the code above in that form. I prefer Python. My blog already covers why so I won’t repeat that. For this purpose, it is simply easier for me to generate the plot of the results in python. You can read or ignore the code at your discretion.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0,10,11)
y = np.array([3.0138902, 1.8036854, 1.0875830, 0.6749870, 0.4167889, 0.2215530, 0.1667404, 0.0906666, 0.0593329, 0.0275748, 0.0046658])

np.polyfit(x,np.log(y), 1)

array([-0.5703016 ,  1.30849592])

plt.plot(x, y, '*', x, np.exp(1.30849592)*np.exp(-0.5703016*x))
plt.xlabel('x')
plt.ylabel('y')
plt.title('Raw data and exponential curve fit of raw data.')
plt.grid()

Figure 1: Curve fit to data.

Strategies for Reducing Cheating Amongst Students

2018-06-15T02:00:00-04:00

The battle between faculty trying to maintain the integrity of their assessments and students trying to gain an unfair advantage has been part of education longer than I've been a part of it as faculty or as a student. It is an eternal challenge that you will win some times, and you will lose others. The tactics undertaken to minimize cheating can be time consuming, sometimes feeling overwhelming, and can negatively impact the rapport between student and teaching and thus jeopardize the educational impact that is the primary goal of education.

With that, I'm going to start with my assessment of the phenomenon of cheating. Further, let me admit that I have often fallen short of my own guidance, but continuously endeavor to do better.

Most students are relatively honest. By relatively, I mean that they will not engage in gross integrity violations, however, they likely will engage in minor, or grey, transgressions. This leads to the first piece of advice: clearly state, on the syllabus and in class, exactly what constitutes cheating and stick to it. Absolute clarity is essential. The most frustrating instances I have faced are ones where the student has a reasonable shred of a claim that they thought that their behavior was acceptable. For instance, if there is no evidence that working with others on homework is considered cheating it cannot be enforced as a violation. Some instructors allow this, some don't. Absolute clarity is necessary. Are they allowed to use internet resources, solutions from other texts, etc.? This is hard! It is normal to have different rules for formative versus summative assignments. Typically rules are more lax for formative assignments to encourage learning through group communication and resource usage. However, in my personal experience it's unwise to have more than two sets of rules for a course. Keep in mind that they are often taking multiple classes and this can look like 10 different sets of rules to them. Simplicity can help preclude unintended violations.
A few students lack any sense of guilt. Various estimates put the percentage of the population who is sociopathic at 1%-4%. Given common teaching loads, that means you likely have at minimum one sociopath in your classes each term. Sociopaths have no sense of guilt and can only be motivated by personal cause and effects that they sense. If they believe they won't get caught, and the rewards are sufficiently high, they will cheat without hesitation. This is one reason why you must enact anti-cheating strategies regardless of everything else. Be cognizant that we all are prone to mirror imaging.
Most students will cheat if the penalty for failure is unbearable. Consider an example: a single parent has spent their last resources on this last class needed to graduate and accept a job that will transform the wellbeing of them and their child, but for whatever reason, they are not expecting to be able to pass the final exam without cheating. Setting aside the arguments that they should have avoided this situation (and I will argue that not all situations can be avoided), the supposition that the student may cheat is very strong. As an instructor or advisor, when one observes such a case, it is prudent to consider advising that the student use the appeals process and another process that can compensate for a justifiable shortcoming of the student (such as a substantial illness). Maintaining academic integrity isn't always easy. There are some who indeed would simply forgive this and look the other way. I believe we have failed as instructors when we reach this point, regardless of the path chosen. The best practice is always to avoid a no-win scenario, about which I will provide some guidance based on my experience and discussions with other faculty.
A scenario that would be a minor stressor for one student can be an immense motivator for cheating for another. While some students will be quite pleased to earn any passing grade in a class, others cannot bear the embarrassing blemish of a non-perfect grade. Cheating may not come from the students who you expect. Their personal expectations are usually not known to the instructor and become hidden motivators.
Cheating students seemingly will invest more time figuring out how to cheat than studying. If often seems that they will spend more time attempting to cheat than it would take to simply perform well in the course. This probably isn't true, but when it is it's likely due to a deficiency in prerequisite material that would necessitate substantially more effort than would be required for a more prepared student.
The risk/reward ratio plays strongly into the decision process of students. If the reward is high (substantive impact on grades), cheating is motivated. If the risk is low, cheating is likewise motivated. Maintaining an appropriate balance that demotivates cheating for the vast majority of students is critical to minimizing this behavior and improve your success.
Good students plead for enforcement of academic integrity. They feel that the value of their degree and GPA is undermined by the false achievements of those who cheat. When you identify these students, listen. They often feel ignored. They want to help you, and they want your help. They often resign themselves to the fact that the situation happens but can be revived if you give them hope that you will act on it.
Not enforcing academic integrity rigorously and thoroughly sends the message that it acceptable. In doing so, an instructor makes themselves a target for cheating and results in loss of respect from students. It also fails in a core tenet of education- the development of ethical graduates.
Preventing cheating is better than catching cheating. It is almost always better to prevent cheating rather than trap students. Building the habit of ethical behavior will hopefully lead to sustained ethical behavior. It won't always, but enforcing academic policies after a violation is always more time consuming than precluding the violation and yet more damage is done to the student, the instructor, and the institution. Thus there is indeed a selfish benefit for the institution and instructor to attempt to preclude cheating. When enforcing academic integrity policies becomes regular, the culture has broken down and needs to be rebuilt, in some part with examples made.
When enforcement is neglected, you have lost. The integrity of your classes is lost and you will have lost the respect of your peers and best students. Enforcement is never easy, but a reputation of strict enforcement will quickly reduce the amount of violations and thus the workload of enforcement. Failure to do so increases this workload.

Homework

Preventing cheating on unsupervised work is the most challenging for obvious reasons. The principle goal of homework is to be formative. In an ideal world no scores would be given but helpful guidance with the instructor learning about the students' progress and adjusting to compensate. However, we know this doesn't work. So, some reward or penalty must be assigned based on performance to motivate sincere effort on the part of students. Too little and the assignments are ignored. Too much and the risk reward ratio motivates cheating. Common recommendations are between 5% and 10% of the grade should be for the regular small assignments that constitute homework. I find that more motivates students to work so closely together that individual performance is correlated too strongly with the best student in a group.

Homework Solutions

Gone are the days when we controlled access to the homework solutions. It's long been time to accept defeat on this matter, although I still comply with publisher agreements not to make solutions available. The internet, high quality cell phone cameras, and social media have resulted in an ecosystem where one must simply accept that access to solutions exists. With that said, there is little risk of getting caught. No peer need ever know- in years past they would have had to share paper copies. There is little risk, or should be little risk. However it turns out that weak students can bungle this up. They often copy solutions verbatim as their laziness precludes them working to even have a basic understanding of the solution. Thus the first piece of advise to preclude cheating.

Require very specific formats and organization for homework. Rigorously enforce them. It should not conform to the format of the solution manual.

Students who cheat will struggle to take one solution and reorganize it to another structure. Further, many solution manuals' solutions are very poorly formatted for a hand written solution. They want for space and the authors typically skip steps for the sake of expediency. What is obvious to an expert who wrote a solution manual is not obvious to a learner. This can be leveraged to always require extra details that require student knowledge to complete.

Further:

Be very clear in writing on the syllabus, and reiterate verbally, what constitutes cheating specific to homework. What is allowed, and what is not allowed. Eliminate any grey areas. They benefit no one.

You will not prevent all cheating on homework. The reward/risk is far too high for some students. You should accept that some cheating is inevitable or you will become a "homework cheating Don Quixote".

You will not identify all cheating on homework. Don't let a lack of observation lead you to believe it isn't happening. However, never make a unsupportable accusation. You will pay a substantial price with more students than just the accused.

Projects/Papers/Reports

Projects are the bane of existence for many students. Better students often hate working with others as it slows them down. Weaker students dislike the exposure of their lack of fundamental knowledge to better students who can readily observe it and disdain them. Students who don't (or didn't) cheat feel aggrieved at perceptions of the ease at which others might cheat through using disallowed sources, upperclassmen, or simply leveraging stronger partners in a group.

The first piece of advice is, when appropriate:

Always subject projects to plagiarism checks. Services such as TurnItIn enable easy plagiarism checking for many projects. Hopefully your institution subscribes to such a service. Eventually they will have a library of projects from your courses to preclude copying from former students.

The second is:

Have students run their own plagiarism checks. There is no need to make this harder on yourself than necessary. When students discover how easy it is for us to identify cheating they stop, with rare exceptions.

When students work on group projects, they naturally form groups that may not be to your liking for pedagogical reasons. This isn't an article about that process. Regardless of how groups are formed, cheating in group projects can be seen as taking credit for other teammates work. It is often overlooked by teachers because a single grade is assigned to the effort with each student receiving the same grade regardless of effort or productivity.

However, there is an easy way to adjust for this.

At a minimum, have students score each other on their performance as a whole, but privately.

My simple process, which works exceedingly well, is that I give each student 100 percentage points to distribute points amongst themselves and their teammates. In my observation substantial value is obtained by having them self-assess.

For Example:

Three students will have 300 points to distribute (each having discretion over 100), 4 will have 400, and so on. The average number of points each student can receive from their peer will thus be 100%. This becomes a score I weigh the project score with to adjust recorded scores up or down. I look for consistency (a member is scored the same by other team members), which is generally very good, and I look for altruism.

Student Prj score Peer weighting Net peer Net score

Ben Sue Jen

Ben 90 35 35 37 107 96.3

Sue 90 25 20 25 70 63

Jen 90 40 45 38 100 110.7

TOTALS NA 100 100 100 100 AVE: 90

In this example, Ben distributed his 100 percentage points 35, 25, 40, to Ben (himself), Sue, and Jen. Clearly he thinks Sue didn't do much, but Jen did more than him. Looking at the table, Ben is consistently thought to be the #2 performing group member, Sue the worst, with Jen the best. Also observable is a trend that is nearly university in my observations: students rate themselves equal to or worse than their peers rate them. Ben's average rating is 36.7, Sue's rating is 23.3, and Jen's is 41. Selfishness is obvious and exceedingly rare.

Student	Prj score	Peer weighting	Net peer	Net score
Ben	90	35	35	37	107	96.3
Sue	90	25	20	25	70	63
Jen	90	40	45	38	100	110.7
TOTALS	NA	100	100	100	100	AVE: 90

In only one single group have I seen an instance where a student scored themselves higher than their peers scored them. Consistently I see students scoring themselves lower than they are scored by their peers. It is a marvelous thing to see.

With this model, students do not complain when they have a weak partner as this usually produces a bump in their grade (providing opportunity for more work and thus recognition in the class). Further, I see stronger students giving more credit to weaker students then the weak students, indicating generosity of spirit. I have seen students form a group strategically in order to leverage this. All students benefitted from the arrangement- the lazy student did little work, but because of the generosity of the strong students, received some credit. The stronger student had more upside available- the points not given to the weak student- and thus improved their grades. An experienced teacher will quickly realize that the high caliber students, while improving their project score, did not increase their score in the class because they had A grades almost regardless of performance on the project. Yet, those students, with this methodology, actively seek out weaker students which benefits those students by being able to observe work habits of stronger peers.

Computer Programs

Proving cheating on programing assignments can be daunting. This happened to me in Fall 2015. I was faced with the challenge of proving that 5 students had indeed written the same code. It is not unusual for successful codes to look identical, so how was I to prove cheating?

In this case I did have one major advantage- none of the students had a program that implemented the assigned algorithm. Further, they all had the same nonsensical "algorithm." As non-working as it was, calling it an algorithm is giving it too much credit.

However, I was sufficiently motivated to make sure that the integrity violation case was successful. Here is what I looked for:

Spacing habits. Individuals writing code will tend to put spaces in some locations, and not in others, depending on their personal habits. Weak students will not follow standards for readability. I won't go further, as I don't want to solve the problem for future students. The effort and skill necessary to cheat well is often unavailable to students who wish to cheat.
Spacing trends. The first equal sign had no space before it, the second did, the third did, the fourth didn't, etc. This pattern was consistent amongst the cheaters... an unlikely scenario.
Indent trends. Indenting is usually done for the sake of readability. When one doesn't understand the code, it's done to convince the grader that something works. This set of students shared meaningless indentation patterns.
Unused variables. Each student defined a variable in the same location that was never used elsewhere with the same values.
Common variable names: this is actually quite common when using conventions. In fact, it's very common. Cheaters use changing variable names to cover up that the code was copied. The 100% lack of shared variable names was suspicious. For instance, t for time is common. One expects commonality, but there was none.
Nonsensical variable names. The desperation to have 5 different names for the same thing lead to arcane naming that didn't fit any convention or logic.
Inability to execute without an error. One code managed to be messed up enough that it wouldn't even execute. However, it did somehow produce a plot for the report. Always run suspicious code
The graphs turned in didn't match the output of the code. Labels were not what the code said they should be. The graph was generated before the code was edited.
Nonsensical settings. The codes shared nonsensical settings that individually wouldn't be surprising. However, the same setting existed in each code. It did nothing except highlight that the code was copied.

Quizzes & Exams

By and large the predominance of the grade of most courses derives from the summative assessments. These present a substantial reward for the student should they successfully cheat. These activities are subject to the greatest strategizing, and faculty, including myself, eventually find themselves impressed by the ingenuity or audacity of students attempting to cheat. This is where student efforts are focused.

Broadly, preventing cheating requires supervision. Supervision requires diligence and constant suspicion of deception.

I recommend an extra proctor for more than 10 students, and an additional proctor for each additional 20 students (31 students requires 3 observers). There must always be eyes available when proctors are busy answering questions as the questions are often decoys to draw the instructor away from a friend.

As an alternative (or better, addition) to more proctors, outfit the room with cameras to record the entire exam. Suspicious behavior that is recorded can be used in integrity hearing. However, in my opinion it's best to:

Make sure students are aware of the cameras, and make a big deal of turning them on. (precluding cheating is best for all involved)
Put the cameras in the back so the students don't know if they are even on (deterrence only requires creating the belief of supervision)

At least one proctor should sit behind the class. A student cannot be sure of safely cheating if they cannot locate all proctors and ensure they will not be observed.

Broad categories of cheating and strategies for preventing them are:

Inappropriately transporting information into an exam

Examples (nonexclusive) are:

Unauthorized crib sheets, which may be hidden:
- Under a piece of clothing
- Inside a ball point pen
- A rubber band (stretch, write, let contract to hide)
- Water bottle (the water facing side of the label)
- Any place, if a student takes a bathroom break
Information already written on nominally blank paper or exam sheets
Information written on the body on the:
- Hand
- Forearm (under a sleeve)
- Ankle (under a sock)
- Other location that can be covered and uncovered as needed
Information stored in calculators or other electronic devices
Information written on an eraser
Information written on the bottom of a shoe

The creativity is unlimited. Actions to prevent this include:

Keep what students may bring to an exam to a bare minimum.
Make an exam open-book and open-notes precludes cheating by bringing these materials from being possible. The fewer the limitations, the less the enforcement. However, be clear about the line.
Be very suspicious of a student who is dressed unusually for them or carrying an unusual item for them. This is obviously easier in a small class.
Reduce the allowable level of calculator, or write problems to remove the value. Modern calculators are quite capable of storing notes and solutions.

Communicating with others

The ability of students to communicate must be managed. Students must, at the very least, have cell phones stored in their bags out of sight. Better yet, these should be set to the side of the classroom.

If there are extenuating circumstances, the cell phone should be given to the instructor who can answer in an emergency. This need often evaporates when this option is provided.

Many modern calculators are cable of communicating through various means. You must become aware of the models being used and their communication ability. Blocking wifi may be allowable during an exam, or traffic may be able to be monitored.

Walking amongst students, and sneaking up from behind (not to startle), is the best way to make sure that they are aware they may be caught using a calculator inappropriately. Fear is the deterrent here. However, being too aggressive can be overly intrusive to compliant students.

A classic cheating strategy is to have a four-person team, although variants can exist. Through a nonchalant action, such as a cough, sneeze, scratch, or tap, a student may alert others of their desire for misdirection. A distant student who is part of the team raises a hand to ask a question. While the proctor is distracted, communication occurs between the originating student and another.

Do not allow students to chose where they sit. This is allowing cheating students to make the first move.

For some students, I will manually assign seating to control appropriate separation. Other times, I will randomly assign seating.

A quick method to assign seats (if they are not numbered) is to write numbers 1-100 (for a class of 100) and have them placed in an organized pattern on the tables. Number exam booklets (I always use exam booklets to control information) and hand them out to students strategically. Students who walk in together do not get consecutively numbered exams. Students who recognize as friends are also intentionally separated.

Cheating off other students' exam papers

This is the classic scenario that we are all aware of. A weak student tries to observe a strong students exam to obtain information.

Preventative measures include:

Change the order of answers in multiple choice problems

Change fundamental numbers in the problem statement and have multiple versions of the exam. Do not change a physical constant: students will remember g = 9.86 m/s^2. Change another number, say the mass. Even better, change a digit. For instance, 13 kg versus 18 kg. From a distance, the numbers look similar. The student who copies may not notice the subtle difference and "solve" with the wrong numbers- a dead giveaway.

Change colors of the exams. While this is a staple and indicator of the exams that are common, that isn't necessarily true. To spice things up:

change colors without changing any problems

change colors, but don't have them correlate with different problem (or multiple choice answer) sets

Students will build their expectations early in the term. Change them on the final exam, and be unpredictable. Do not enable them to prepare to counter your move. It isn't the most ambitious students who are likely your concern, but they will spend a small amount of time strategizing.

Take-home exams

Take-home exams create a massive challenge in that the reward is great and the probability of getting caught is minimal. In part you will be relying on weak students doing such a poor job cheating that they are easily caught when they could have easily avoided detection.

Given the ease with which students can cheat on out-of-class activities, I advise against them, although some classes make it difficult to avoid. Classes where this is hard to avoid are those where solutions are too long for a reasonable exam period. Small classes are easier to manage, but still trustworthiness is suspect. Do not get fooled into thinking you know if they cheated or not from what they turn in. It's very difficult.

When a take home exam is given, test a portion of what should have been learned/understood and test it in-class or one-on-one. This provides some validation that the student at least understands what they turned in.

Carefully scrutinize patterns in turned solutions for indications of non-original thought. Here you must rely on exactly the opposite structure for homework: do not indicate the structure of the solution and organization. You will use improbable consistencies to indicate collusion.

Slight changed to problems are prudent. Leverage the reality that only strong students can generalize solutions, and they will be unwilling to share their work. Weaker students are less likely to be able to generalize and successfully work together or translate one solution to another.

Further, attempt to generate problems where the solution can follow separate paths- a bifurcation of sorts. For instance, in fluid dynamics, calculation of a Reynolds number results in an a determination of how to proceed with the problem. The entire path is different, resulting in solutions that look very dissimilar from the students' point forward.

Summary

This is the most distasteful part of the job. Nobody likes it. However, with a bit of experience and prudence, your effort can be minimized and the incidences greatly reduced. Please provide your feedback below and I will incorporate it as I receive it.

Why I Moved to Python From Proprietary Solutions and Will Not Go Back

2018-03-15T23:21:00-04:00

Three years ago I wrote my first ever blog entry: Why Matlab Users Should be Moving to Python. It opened with:

Back in 1989, when I first entered graduate school as an engineering student, I knew a handful of languages to some modest degree. For the sake of computations, I knew Basic and Fortran 77 (the horror!).

My knowledge of programming was elementary by the standards of the day, but I did fancy myself as relatively capable. Of course, I made that judgement based on the company I kept, and Matlab was plenty powerful to perform the tasks we needed (still is- even more so). Amongst engineers in my field I was close to the top in "getting stuff done" computationally and still am. However, that came at the price of not developing the next level of high-power computing skills and I lived for 25 years in the golden cage of a single commercial software solution. If they didn't have it, I didn't know about it. It continues to grow and thrive, but there is far more growing and thriving that that singular environment.

As a result I'm doubling down on open source, with Python as my primary tool. Why?

I'm an engineer. My time will always be focused on engineering and the time to create great C or Fortran that is parallelized for a particular problem won't happen unless it's necessary. It's a lot of work- a task in itself. If I need to, I'll do it, but it's not a great starting point because the time spent programming exceeds time spent executing code far too often. Code should be written as minimally as possible to get the job done. The best code is no code at all. C and Fortran are not concise and require details that interpreted languages do not. There is a benefit to those details, but most of the time they bear little real fruit.
The best code is no code at all. Code is our enemy. That means libraries, libraries, libraries, or in Python languages packages (soon to be here). Python had 135,356 projects at the moment I wrote this, and growing. The open source nature, amongst other benefits, has made Python the place to share your package. As a user, that means we get to leverage all of this instead of writing the code all over again. Matthew Rocklin's blog opens with an outstanding observation: Programming often limits research more than fundamental domain challenges.. The Python ecosystem continues to grow and be put together to minimize how much coding users have to do themselves.
Open source means people gladly help you do your work. When you make your project open source and post it, as I do on GitHub, and then announce it, you might think "oh, I'm just going to embarrass myself." False! I will posit that humility is an effective way to grow. Setting that aside, what you will find is others who are interested in your work. They won't usually take over and do it for you, but sometimes they will as Raphael Timbó did with the Vibration Toolbox for Python. [1] Short of that, they will provide fixes, highlight deficiencies, provide documentation, etc. Many will test your code for you, many will provide programing advice to improve your programing skills.

Getting things done takes priority over writing great code. Unfortunately, this results in a lot of code, often poorly documented, with limited reusability (especially by others). Further, this habit creates a culture of isolation, where I and hundreds of others, perhaps thousands, have rewritten the same code for nearly the same purpose, without thousands if not tens of thousands of hours wasted and who knows how many incorrect results impacting designs. The argument for using commercial code is usually something like: * well, let experts write code for you. They validate it, and you can trust it as the best that can be.* Well, that's simply wrong.

First, consider the warrantee you agree to when you purchase and use commercial code. They don't take any responsibility for the code being fit for your purpose. That's on you!
Second, major corporations validate commercial code then validate it more. Then, they revalidate it. They take the attitude trust but verify, which means they don't trust it, but find it more cost effective than writing their own code.

How do I know this? My job is to know and talk to people in industry regarding research, technology, and our graduates and their preparation. More o, they are writing their own code as the cost savings isn't as grea

With about an hour to go a fellow graduate student, Jeff Umland- now of JPL, suggested that perhaps I should heed his advice and use Matlab. [2] The project would be simple in Matlab according to him. There was certainly no way I could complete the project in Fortran 77 in the remaining time. So, with him responding to my shouted out questions (learning under fire - ''How do you write a loop?'') I started using Matlab and finished with time to spare. That was the last time I started a project using Fortran. Fortran has its place (although it seems to be slowly fading away), but it has been clear to me ever since that for most problems facing me, and most engineers, time of development far exceeds time of program execution. When it doesn't, your prototype in a scriptable language is still not a bad first start. So, the speed cost of using an interpreted language was well worth the savings in development time.

Unfortunately, this can build some bad habits that I am still working through. Nevertheless, I continued to learn Matlab and later wrote the Engineering Vibration Toolbox in Matlab. I immediately noticed that in Matlab each function is the name of a file, an once that file is in the path, you can have a clash of names (namespaces are a major improvement with Python). Over the years many of my programs have been broken because of the shared name space, which continues to grow. Names such as addpath, spectrum, plot3, etc. have been replace by Mathworks m-files. Further, when I distribute a code, I have to be concerned about their clashes. Fortunately I was savvy enough to effectively create my own name space for the Engineering Vibration Toolbox to name each and every file vtb#_#.m, hoping that my starting them with vtb they would remain safe. That seems to have stood the test of time, but no function has a name that can be reasonably guessed. They just have ugly serial numbers.

So, although the names stood the test of time, many of my programs written for personal (not for distribution) use did not. I've become further disenfranchised by the lack of ability for me to truly own my own codes. Without a Matlab license, until Octave came around, your M-files were of little value if you didn't continue to pay the MathWorks for a license. I appreciate the effort that they put into writing Matlab, but I put time into my code as well and I am not willing to be hostage to them. My code doesn't exist without me paying the MathWorks to use it. That means it's not truly my code.

As some might guess, I later switched to Octave. Ironically, what I liked the best about Octave initially was that it wasn't a Matlab clone, but nominally equivalent for most tasks. As time has progressed the author, John Eaton, has acquiesced to public pleading and moved further and further towards Matlab compatibility. [3] While I respect why he made the move, for my interests it downgraded Octave, although I'm likely a minority, but it is likely the right decision for the Octave ecosystem.

I still love Octave and have great respect for the John (he is certainly a much more accomplished programmer), but I noticed Python and that it seemed to really be taking scientific programming to another level after Travis Oliphant created NumPy. The add-on packages for scientific programming are extensive. The language isn't designed to be purely for computation/data manipulation, but in many respects, it's better, IMHO. Instead of rehashing arguments, I'll redirect you to the opinions of Pyzo, Hoyt Koepke, and Phillip M. Feldman. Further, as scientific support for Python continues, the speed penalty compared to a compiled language is disappearing. I personally love the readability of the code, and that prevents mistakes. It just looks good. PEP 8 and other standards can seem onerous to new users, but in time they prove their value in generating code that is more easily readable and debuggable. Installing scientific tools is easy with modern distributions documented at NumFOCUS, and other packages (including my own) can be easily installed from PyPI using PIP.

I will likely use them (Matlab/Octave) again in the future. However, Python has empowered me to generate more capable code than I ever could in Matlab due to Matlab's limitations and focus. Python's malleability for such a wide variety of tasks has made it the best single language to learn, IMHO. It will often not be the best language for a single task. However, it is good enough often enough that if you are going to learn a single language, Python is your best bet for your multi-tool. If you have time to learn multiple languages, and have specific needs, by all means grow your language skills as necessary. However, if you spend little time programming, Python is likely the most power you can get for the time you have available.

Incidentally, and I made this blog in Python, using Pelican, writing the source files in ReST.

[1]	I have always been a bit stubborn in my opinion. I think I'd made some positive strides as I've grown older.

[2]	Ironically, early on he observed clear reluctance on my part to convert to the Octave standards of Octave's help system compared to Matlab. I think he missed the moment when I came to agree with him about that and other things! I'm sorry!

[3]	In fact, he drove me crazy pushing the project forward so fast that I spent extra time working on formulating project objectives and priorities. In a few months we had a working release, and the project has had thousands of downloads since.

kwargs, args, **kwargs, and *args

2017-12-03T07:28:00-05:00

In my observation, nothing in Python is more confusing than kwargs and args and how to use them. There are explanations on the web, but none ever seem comprehensive enough for me. So, here is my attempt. Like anything else, they are a feature that seem to have no point until you are stuck needing them. Then they become invaluable.

(This notebook can be downloaded and run within the Jupyter notebook environment by selecting the link and using Save Page As in your web browser.)

They don’t exist

What? How can they not exist? Well, the Python only reserves a limited number of keywords. 33 to be exact. args and kwargs aren’t on the list. kwargs and args are variable names you choose.

If they don’t exist, why do I see them all over the place?

They have become conventional variable names for packed arguments and packed keyword arguments. You could use any other name you’d like that is allowable in Python. These are just conventions.

What is packing?

Packing is collecting a group of values into a single variable name. * collects values into a tuple while ** collects named values into a dictionary. The distinction is somewhat irrelevant for the purpose here, outside of the apparent bundling of name and value in the dictionary.

* and ** are actually used outside of args and kwargs. For instance, some functions return multiple values. They can be packed by calling them with a single variable, e.g. a = f(x) in place of a, b, c = f(x). Alternatively, if you only need the first value, you can instead call the function with a, *_ = f(x), which packs the rest of the returned values into _, which represents nowhere.

How does `*args` work in a function?

Consider a function into which we want to send a variable number of arguments.

def f(*args):
    print(args)

f('This', 'is', 'how', 'the', 'print', 'command', 'works.')

('This', 'is', 'how', 'the', 'print', 'command', 'works.')

Of course, this isn’t quite right. What this shows is that all of the arguments have been assembled into a tuple. They now are part of a list, and print as such. This isn’t likely what we meant to do. What we want to do now is unpack the tuple so that the print command perceives the tuple as a group of individual arguments.

def g(*args):    # packed into the tuple
    print(*args) # unpacked into individual arguments

g('This', 'is', 'how', 'the', 'print', 'command', 'works.')

This is how the print command works.

Now, the Python print command is much more sophisticated, but this starts to show the power of packing arguments this way.

What is an example where this could be helpful?

Say I want to write a function that returns the product of a number to the 3rd power, such as

\begin{equation*} a b^3 \end{equation*}

However, I already have a more general function that returns

\begin{equation*} a b^c \end{equation*}

def power(a, b, c):
    return a*b**c

power(5, 2, 3)

What I can do is create a function that calls power, but packs and unpacks the arguments appropriately. We know that the last argument must be the number 3. We could simply explicitly write the other arguments:

def power3(a, b):
    return power(a, b, 3)

However, that’s not as lazy as we would like to be. We could have instead written:

def power3(*args):
    return power(*args, 3)

power3(5, 2)

Why would this be helpful? Well, there are numerous reasons. One might be error checking. It may be that power is from a module that is very powerful so you want to use that. However, your code may call it with arguments that are inappropriate for that library. Here I will use a string, but you can imagine that perhaps a complex number may not be allowed. So, we can write:

def power3(*args):
    if type(args[0]) is str:
        print('Oops. You sent me a string.')
        return
    else:
        return power(*args,3)

power3(5, 2)

power3('Hello')

Oops. You sent me a string.

Note that the string was packed into a tuple of length 1 so the code must access the 0th value.

The key here is the *, not the name args. Any variable name could have been used.

So what about `**kwargs`?

This is the same situation, but this time with named variables and dictionaries. This time consider writing a central finite differences operator such that

\begin{equation*} \frac{df}{dt}:=\frac{f(t+\Delta t/2)-f(t-\Delta t/2)}{\Delta t} \end{equation*}

However, if we want this to work for any function returning a numerical value, but with an unknown set of unnamed and named arguments, we can abstract by using *args and **kwargs:

def diff(f, t, dt, *args, **kwargs):
    return (f(t+dt/2, *args, **kwargs)
            -f(t-dt/2, *args, **kwargs))/dt

f is the name of the function we will calculate the slope of, t is the variable we want the slope with respect to, dt is a step size for that derivative. All other arguments, named or not, at passed directly through to the function f.

Let’s define our power function again.

def power(t, coeff=5, exponent=3):
    return coeff*t**exponent

Next we obtain the slope when \(t=2\), with a coefficient of 2 and exponent of 3.

diff(power, 2, 0.001, coeff=2, exponent=3)

24.012001999995647

This can work with any other similarly formed function. This is a simple product of the three arguments, 2 unnamed, one named.

def product(x, y, z=3):
    return x*y*z

We obtain the slope at \(x=2\) with \(y=4\) and \(z=3\).

diff(product, 2, .01, 4, z=3)

11.999999999999744

The keys for using **kwargs is to remember: - kwargs is your variable name. You decide what it is. - kwargs is now a dictionary. If you want to use them to call another function, make sure to unpack it in the function call by using **. - You can access parts of kwargs just as you can any other dictionary.

Conclusion

Hopefully this at least clarifies why they are useful and how you can apply them. In my experience, they are most useful when a function is designed to call a future (as yet defined) that requires parameters that can not be anticipated. Regardless, please leave your comments. I hope this helps.

Using Laplace Transforms to Solve a Linear Differential Equation in SymPy

2017-05-06T11:21:00-04:00

Consider solving

\begin{equation*} 3\ddot{x}+30\dot{x}+63x=4\dot{g}(t)+6g(t) \end{equation*}

in Jupyter where

\begin{equation*} g(t)=u_s(t) \end{equation*}

is the unit step function (Heaviside Function) and \(x(0) = 4\) and \(\dot{x}(0)=7\).

%matplotlib inline
 from sympy.integrals.transforms import laplace_transform
 from sympy.integrals.transforms import inverse_laplace_transform
 from sympy import *
 import sympy as sympy
 from sympy.abc import a, t, x, s, X, g, G
 init_printing(use_unicode=True)

x, g, X = symbols('x g X', cls = Function)

x0 = 4
 v0 = 7

g = Heaviside(t) #This is just the step function

laplace_transform(Heaviside(t), t, s) # Checking the performance of the Laplace Transform Module

(1/s, 0, True)

laplace_transform(DiracDelta(t), t, s) # Checking the performance... this is wrong. Should get 1.

(1/2, -oo, True)

laplace_transform(DiracDelta(t - 2), t, s) # right result

(exp(-2*s), -oo, True)

laplace_transform(DiracDelta(t - a), t, s)[0] # right result (theta(t) is a Heaviside function representation in SymPy)

\begin{equation*} \left(- \theta\left(a e^{i \pi}\right) + 1\right) e^{- a s} \end{equation*}

laplace_transform(DiracDelta(t - a), t, s)[0].subs(a,2) # Looks good

\begin{equation*} e^{- 2 s} \end{equation*}

laplace_transform(DiracDelta(t - a), t, s)[0].subs(a,1) # Looks good

\begin{equation*} e^{- s} \end{equation*}

laplace_transform(DiracDelta(t - a), t, s)[0].subs(a,0.0000001) # Looks to converge

\begin{equation*} e^{- 1.0 \cdot 10^{-7} s} \end{equation*}

laplace_transform(DiracDelta(t - a), t, s)[0].subs(a,-0.0000001) #So... here is the illustration.

\begin{equation*} 0 \end{equation*}

What's happening here is that SymPy currently takes the position that half the Dirac delta happens before zero, half after, so the result should only be half as big. I think I'm in a rather large crowd in saying that this isn't proper. It makes some theoretical sense, and is a wonderful math debate. However, given convention says that \(\delta(t)\) is fully captured by a Laplace transform with a result of \(1\) (Mathematica, Maple, Matlab, every System Dynamics, Controls, and Signal Processing book I've ever read), SymPy is practically wrong. I'm hoping that they will change their minds. I am now a bit skeptical about using SymPy for my math work as the results of a simple conventional application don't match expected behavior consistent with other codes.[1]_

So we note that SymPy isn't taking the Laplace Transform properly here, so we need to avoid using this result. (we should have gotten 1) Valid as of 0.7.6.1

print(sympy.__version__)

0.7.6.1

The first line below would work if SymPy performed the Laplace Transform of the Dirac Delta correctly. Short of that, we manually insert the Laplace Transform of \(g(t)\) and \(\dot{g}(t)\) where \(g(t)=u(t)\).

#eom  = Eq(3*(s**2 * X(s)-s*x0-v0)+30 * (s*X(s)-x0)+ 63 * X(s), laplace_transform( 4 * diff(g,t) + 6 * g, t, s, noconds = True))

 eom  = Eq(3*(s**2 * X(s)-s*x0-v0)+30 * (s*X(s)-x0)+ 63 * X(s), 4 * 1 + 6 * 1/s)
 eom

\begin{equation*} 3 s^{2} X{\left (s \right )} + 30 s X{\left (s \right )} - 12 s + 63 X{\left (s \right )} - 141 = 4 + \frac{6}{s} \end{equation*}

Xofs = solve(eom,X(s))
Xofs[0]

\begin{equation*} \frac{12 s^{2} + 145 s + 6}{3 s \left(s^{2} + 10 s + 21\right)} \end{equation*}

soln = inverse_laplace_transform(Xofs[0],s,t)
soln

\begin{equation*} \frac{\theta\left(t\right)}{84} \left(8 e^{7 t} + 749 e^{4 t} - 421\right) e^{- 7 t} \end{equation*}

soln_simp = expand(soln)
soln_simp

\begin{equation*} \frac{2 \theta\left(t\right)}{21} + \frac{107 \theta\left(t\right)}{12} e^{- 3 t} - \frac{421 \theta\left(t\right)}{84} e^{- 7 t} \end{equation*}

N(soln_simp,5)

\begin{equation*} 0.095238 \theta\left(t\right) + 8.9167 e^{- 3 t} \theta\left(t\right) - 5.0119 e^{- 7 t} \theta\left(t\right) \end{equation*}

Note that \(\theta(t)\) is SymPy's notation for a step function. This simply means the answer can't be used before \(t=0\).

[1]	This has been fixed in later revisions.

Installing Python for Scientists and Engineers (and maybe Mathematicians too)

2017-01-13T11:00:00-05:00

The title of this article implies that you have similar interests to me: computational usage. I'm going to try to address 99% of the computational community so there are parts of this page you won't need. I apologize. However, I find it valuable to have a single destination to point people to when I have a question like "how do I start using Python?".

Python is quickly growing acceptance in the scientific and engineering community as a computational language due to its open source nature, meaning it can be installed anywhere for free, and its quickly expanding toolkit. With the scientific packages being developed at a rapid pace it is quickly approaching parity with established commercial tools such as Matlab, or it's free (and extremely well developed and supported) surrogate Octave, and even exceeding it in some domains. My reasons for moving to Python have previously been discussed in My Journey to Python and continue to grow as I discover greater power in using it for my primary language. My other getting started entry may also appeal to help individuals although the Plotting with matplotlib is likely the best way to start from zero knowledge.

The question I often hear, though, is how do I try it? How does someone install Python? The answer is simple, once your needs are known. It's simple because many smart people have worked together to make it simple. Doing this as an individual would be prohibitively time consuming and fail to leverage tens of thousands of hours spent solving problems already. So: you don't want to do that. Not now at least. Probably not in the future either. It's not something to do unless you have a lot of spare time.

What is everything, and why you don't want to manually install it

There is no reason to read this subsection if you trust me and are raring to go. I'm the kind of person who would read this section, so that's why it's here. You might want to look at it later.

Just to give a flavor of why you wouldn't want to do this, note that the installation I recommend installs 445 unique sub-packages and ensures that they all work properly, without operating system hangups, missing sub parts, failed compiles of executables, etcetera. These sub-packages include high performance optimally compiled libraries that you would need to install yourself after installing compilers and making sure you have all of the necessary dependencies. If this sounds like a lot of fun, you are reading the wrong page, and I can't even direct you to a right page.

Any platform (Mac/Windows/Linux)

At this point in time, I believe that it is very clear that the hands-down winner for installing Python as a novice is the Anaconda distribution by Continuum Analytics. Their website has plenty of information that you can peruse for getting an installation of everything you will need. The CEO, Travis Oliphant, is widely credited for molding Python into a computational language through his development of Numpy, which became the basis for so much more. While arguments are sometimes made that using a Continuum Analytics is insufficiently "pure" for the open source community, I strongly disagree. [1]

Other available distributions, that I don't personally have experience with, are

1. Active State Python is a Commercial distribution with a "community edition". Active State is a solid supporter of the broader community by being a sponsor of the Python Software Foundation.

2. Enthought Canopy Python. Reviews for this version of Python seem not to be as good as those for the Anaconda version. Enthought was slow to move to Python 3, and it may still not be here (I say here, because we are here!). I don't see the point of a new user starting on a language being phased out (although it seems old languages never die).

3. Python(x,y) I'm not a fan of this as it is still not using Python 3. Due to inertia and some of the revolutionary improvements to Python 3, many with codebases have been slower to adopt Python 3 (now at 3.6!). For those doing scientific computing, the new features now clearly motivate one to start with Python 3. Overcoming legacy inertia can be a substantial challenge, though, so I don't judge anyone stuck on 2.7.

Windows

One the neatest distributions fits wholly on a thumb drive. The WinPython is completely portable an can can be moved to any location (local, network, USB drive). This means you can take it with you, give it to a group of people pre-configured, distribute it over a network, etc. It can be completely "non-invasive", meaning you don't need to even uninstall it. It's not even on your hard drive. This is a great tool for anyone who has to work at different computers on a regular basis.

Mac

I personally have had luck using MacPort, but just to get Python running, it's a bit much. Other package managers for free software exist, including Fink, Homebrew. These all do more than you are likely interested in, and choosing one is far from a simple task (I've use each at one time or another, and they all seem to work fine.)

That said: If you want to install Python and run Python today, use the Anaconda distribution.

Linux

There are really two types of Linux users:

I want a cheap operating system, or one that will still run fast on old hardware
I'm hard core. I would program in assembly language but I don't have enough time.

If you are type 1, install the Anaconda. If you are type 2, God bless you, and good luck. You may have luck using the linux package managers (specific to your distribution), but I will simply wish you well. You're not my target audience.

Government labs

I've a bit of experience here, but not with Python explicitly, and not in any way that would make me worth kidnapping.

Working with anything in a SCIF can be daunting. If you work in one, you know that. If you don't, you only are reading this to get a bit of help convincing your tech people that you should be able to get this installed.

The big argument about open source software is that "anybody" can embed code in the software, risking the security of your systems.

Yup. That's true. However, it's also irrelevant. Here's why. Every single unix system contains open source software on it. Yes, that includes the Mac, which comes with an old underwhelming version of Python. Windows is gradually becoming more open source as well. Why? Because over time we've begun to realize that closed source software is more vulnerable than open source software because vulnerabilities are found faster, and there are more good guys looking than bad guys. The beauty in open source software is that you, the user, can look for vulnerabilities yourself. You may not be qualified, but surely your IT person is (unlikely, but they often act that way by espousing security mantras that have been refuted.).

What about the risk of foreign nationals embedding code in the software? Well, couldn't that be happening to Windows/MacOS/Linux right now? Do you think they don't have programmers oversees working on their operating systems? Call tech support and find out. Commercial packages that are core also have foreign nationals working on them, certainly many on H1B visas. This is also a red herring. The problem with security is it's never as easy as the security people try to tell you it is with platitudes of rules and more rules. Most real vulnerabilities are of the human type, phishing for example. It's getting the door ajar that is the real threat.

So, I'm not an expert in Cyber security, but security people shouldn't make it easy for me to prove them wrong. However, this document won't convince them. So, instead, point out that there is a process already in place to get open source software in a secure environment: get commercial support. Further, you can have them contact other SCIFs and ask them how they are using open source. It's inconceivable that Lawrence Livermore is writing open source software and not using it where it needs to be used. What may actually convince some of them is that open source software is actually safer than commercial software. This is illustrated by the Coverity Scan report which shows that Open source software had a defect density of 0.61, compared to commercial software’s 0.76. [2]

That said, there are valid reasons not to use a dynamic language for code that must be secure. However, one has to ask: is this what's really going on? Are you using Python for analysis, or for delivering code? Those are two different issues that very well may lead to two different security responses.

To specifically address at least one instance: Enthought Canopy at the United States Air Force Research Lab is already approved and ready to be installed on almost any network including RDT&E, enclave networks, and AFNet, but don't trust me. Look it up yourself!

Questions?

These tips only get better with your feedback, so please provide it below.

[1]

The Anaconda distribution you install is completely open source. They constantly feed their software into the open source community, for free, and are unbelievably good citizens in the free software community. Keep in mind: the people doing this work need to eat. They've somehow managed to find a way to give away the product, even provide support for it, and get paid for yet a higher level of service. From my experience, they must flying in to your city, stopping by your MD's waiting room, and helping you code while you wait. I don't know what they do, but they do it, so I don't but this argument. A more eloquent argument is provided by Jake VanderPlas.

[2]	Tackling Open Source's Security Problem.

Scientific Python Understandings: Things I want to remember

2016-12-29T11:21:00-05:00

Part of the fun I've had with Python is watching (and trying to participate in) the rapid development of it's capabilities. I've packaged a couple minor things, array_to_latex and vibration_toolbox, and am working on the Vibration Testing Toolbox or whatever I may name it. You can see I'm still a bit Matlab fixated. However, as I grow, I'm constantly discovering that my Matlab fixation has caused my programming skills to atrophy. So, with my move to python I'm also learning the Python ecosystem who the "players" I need to watch are. Sometimes it's a person, sometimes it's a particular website or blog of interest. Nevertheless, I'm listing them here so I can find them, and maybe that will help you too. The best description of why a link/person is here is that I need to keep going back and looking/watching.

People

Over time I've noticed I keep going back to the same people for help/guidance. Some of them know me, or at least they've responded to my contacts. Others don't have a clue that I have found their work immensely helpful. They are in no specific order.

Travis Oliphant: Scientific computing in Python wouldn't be the ecosystem it is without him.
Jake VanderPlas: A number of great tidbits on high powered computing, plotting, and understanding the Python ecosystem.
Lorena Barba: I also link to her github page below. I don't think she sleeps.
Steven Silvester: Helped me a lot, and has a lot of neat tricks in his pages and packages. Super nice guy.
Olga Botvinnik: Not blogging as much as she once did, but always good reading. Her blog helped me get started in matplotlib. I do wish she had more time to write on Python as she excels in clean and clear explanation.
Roshan Rush: A slew of great YouTube videos on things Python

Pages

The following are pages I've found immensely useful.

Scipy: Central to scientific computing in python
CFD Python
Conda Full Install: People just need to find this often enough

[1]	Ironically, early on he expressed clear reluctance on my part to convert to the standards of the then substantially differing Octave help system. I think he missed when I came to agree with him about that and other things!

A Quick Introduction to Sympy

2016-10-25T09:55:00-04:00

This is a quick demo of Sympy, partly for you, but mostly for me. The actual notebook is available on my github iPythonExamples repository.

# import symbolic capability to Python
from sympy import *

# print things all pretty
from sympy.abc import *
init_printing()

Consider solving the problem

\(y = \frac{x^2\sqrt{3x-2}}{(x+1)^2}\), find \(\frac{dy}{dx}\)

# Need to define variables as symbolic for sympy to use them.
x, y= symbols("x, y", real = True)

# Define y
y = x**2*sqrt(3*x-2)/(x+1)**2
y # This causes it to output formatted nicely. The print command formats it as computer code

\begin{equation*} \frac{x^{2} \sqrt{3 x - 2}}{\left(x + 1\right)^{2}} \end{equation*}

# Taking the natural log of this:
a = ln(x**2*sqrt(3*x-2)/(x+1)**2)
a # Keep in mind that the left hand side is ln(y)

\begin{equation*} \log{\left (\frac{x^{2} \sqrt{3 x - 2}}{\left(x + 1\right)^{2}} \right )} \end{equation*}

# Taking its derivative with respect to x
# So, we took the log of the right side, then the derivative
diff(a,x)

\begin{equation*} \frac{\left(x + 1\right)^{2}}{x^{2} \sqrt{3 x - 2}} \left(\frac{3 x^{2}}{2 \left(x + 1\right)^{2} \sqrt{3 x - 2}} - \frac{2 x^{2} \sqrt{3 x - 2}}{\left(x + 1\right)^{3}} + \frac{2 x \sqrt{3 x - 2}}{\left(x + 1\right)^{2}}\right) \end{equation*}

# That is kind of messy. Let the computer simplify it
simplify(diff(a,x))

\begin{equation*} \frac{3 x^{2} + 15 x - 8}{2 x \left(3 x^{2} + x - 2\right)} \end{equation*}

# We could have done this all at once
simplify(diff(ln(y),x))

\begin{equation*} \frac{3 x^{2} + 15 x - 8}{2 x \left(3 x^{2} + x - 2\right)} \end{equation*}

Now, the left hand side processed the same way is

\begin{equation*} y \end{equation*}

take the natural log

\begin{equation*} \ln(y) \end{equation*}

Take the derivative

\begin{equation*} \frac{1}{y}dy \end{equation*}

So, our entire processed equation is

\begin{equation*} \frac{1}{y}dy = \frac{3 x^{2} + 15 x - 8}{2 x \left(3 x^{2} + x - 2\right)} \end{equation*}

so all we have to do is multiply the processed right side by \(y\) and we have our answer and our derivative is

simplify(diff(ln(y),x)*y)

\begin{equation*} \frac{x \left(3 x^{2} + 15 x - 8\right)}{2 \sqrt{3 x - 2} \left(x^{3} + 3 x^{2} + 3 x + 1\right)} \end{equation*}

Of course, Sympy is capable enough that we can just take the derivative directly

simplify(diff(y,x))

\begin{equation*} \frac{x \left(3 x^{2} + 15 x - 8\right)}{2 \sqrt{3 x - 2} \left(x^{3} + 3 x^{2} + 3 x + 1\right)} \end{equation*}

but that was way too easy.

We could compare them, just to be sure they are the same

simplify(diff(y,x))==simplify(diff(ln(y),x)*y)

True

Or plot this monstrosity

plot((diff(y,x),(x,0.2,10)), (y, (x,0.5,10)))

# To change colors
# show = False delays the plot until we can set all of the parameters
# legend turns on the legend and uses the labels we have later.
p = plot((diff(y,x),(x,0.2,10)), (y, (x,0.5,10)), show = False, legend = True)
p[0].line_color = 'blue'
p[0].label = '$\\frac{dy}{dx}$'
p[1].line_color = 'green'
p[1].label = '$y$'
p.show()

What if we wanted to make a substitution?

y.subs(x,alpha)

\begin{equation*} \frac{\alpha^{2} \sqrt{3 \alpha - 2}}{\left(\alpha + 1\right)^{2}} \end{equation*}

Getting cURL to work in Cygwin

2016-03-01T14:00:00-05:00

Cygwin is a great option for Windows users who want to run Linux software. However, it doesn't always work "right." In my case, trying to install packages in Octave from repositories on GitHub simply would not work. The root cause seems to be that curl wouldn't connect for a variety of reasons, one being the inability to ''resolve the host'', the other being the inability to follow redirects (which I wasn't aware of happening for a very long time)

Two options came to solve the issue.

The -4 option, which helped resolve the host
The -L option, which enables curl to follow redirects.

With that, the call

curl -4 -L https://github.com/vibrationtoolbox/ovtoolbox/archive/master.zip > vtoolbox_temp.zip

works and can be used in the Octave system command.

Solving multiple linear ordinary differential equations in SymPy

2016-02-29T04:37:00-05:00

I am using Python 3.5 in Jupyter (formerly iPython). The original notebook is available at my github examples repository.

Presume we wish to solve the coupled linear ordinary differential equations given by

where \(u(t)\) is the step function and \(x(0)=5\) and \(y(0) = 10\).

%matplotlib inline

# import symbolic capability to Python- namespace is a better idea in a more general code.
from sympy import *

# print things all pretty
from sympy.abc import *
init_printing()

Need to define variables as symbolic for sympy to use them.

t, C1, C2= symbols("t C1 C2")
x, y =  symbols("x y", cls = Function, Function = True)

First we must define the governing equations as equalities or expressions. We have the option, I will do one each way for the sake of illustration, as opposed to simplicity.

eq1 = Eq(3 * diff(x(t), t), y(t))
eq1

\begin{equation*} 3 \frac{d}{d t} x{\left (t \right )} = y{\left (t \right )} \end{equation*}

I'd like to use the Heaviside function. It doesn't work, but I'm showing how I tried anyway.

import sympy.functions.special.delta_functions as sfuncs

eq2 = Eq(diff(y(t),t), - 3 * y(t) - 15 * x(t) + 4 * sfuncs.Heaviside(t))
# Note: Heaviside doesn't work for me in the solver, so I've left it here as "proper" but dokn't use it.
eq2 = Eq(diff(y(t),t), - 3 * y(t) - 15 * x(t) + 4 * 1)
eq2

\begin{equation*} \frac{d}{d t} y{\left (t \right )} = - 15 x{\left (t \right )} - 3 y{\left (t \right )} + 4 \end{equation*}

Solving the differential equations. The ics =... should apply the initial conditions. Doesn't work. Perhaps some day.

soln = dsolve((eq1, eq2), ics = {x: 5, y: 0})
soln

\begin{equation*} \left [ x{\left (t \right )} = \frac{1}{3} \left(C_{1} \sin{\left (\frac{\sqrt{11} t}{2} \right )} + C_{2} \cos{\left (\frac{\sqrt{11} t}{2} \right )}\right) e^{- \frac{3 t}{2}} + \frac{4}{15}, \quad y{\left (t \right )} = \left(\left(- \frac{3 C_{1}}{2} - \frac{\sqrt{11} C_{2}}{2}\right) \sin{\left (\frac{\sqrt{11} t}{2} \right )} + \left(\frac{\sqrt{11} C_{1}}{2} - \frac{3 C_{2}}{2}\right) \cos{\left (\frac{\sqrt{11} t}{2} \right )}\right) e^{- \frac{3 t}{2}}\right ] \end{equation*}

Solving for the constants. I'm substituting t = 0, then the initial values for x and y. You'll note that the first equation doesn't need y(0) substituted, while the second doesn't need x(0) substituted. I found this out after and did that for brevity.

constants = solve((soln[0].subs(t,0).subs(x(0),5), soln[1].subs(t,0).subs(y(0),10)),{C1,C2})
constants

\begin{equation*} \left \{ C_{1} : \frac{313 \sqrt{11}}{55}, \quad C_{2} : \frac{71}{5}\right \} \end{equation*}

Let's put in our constants and see what we get. I'm using .rhs to pull out the right side of the solution. You can look at soln[0] to see what I mean, or try help(soln[0]) and read the results.

xsoln = expand(soln[0].rhs.subs(constants))
xsoln

\begin{equation*} \frac{4}{15} + \frac{313 \sqrt{11}}{165} e^{- \frac{3 t}{2}} \sin{\left (\frac{\sqrt{11} t}{2} \right )} + \frac{71}{15} e^{- \frac{3 t}{2}} \cos{\left (\frac{\sqrt{11} t}{2} \right )} \end{equation*}

ysoln = soln[1].rhs.subs(constants)
ysoln

\begin{equation*} \left(- \frac{172 \sqrt{11}}{11} \sin{\left (\frac{\sqrt{11} t}{2} \right )} + 10 \cos{\left (\frac{\sqrt{11} t}{2} \right )}\right) e^{- \frac{3 t}{2}} \end{equation*}

eq1.subs(x(t),xsoln).subs(y(t),ysoln)

\begin{equation*} 3 \frac{d}{d t}\left(\frac{4}{15} + \frac{313 \sqrt{11}}{165} e^{- \frac{3 t}{2}} \sin{\left (\frac{\sqrt{11} t}{2} \right )} + \frac{71}{15} e^{- \frac{3 t}{2}} \cos{\left (\frac{\sqrt{11} t}{2} \right )}\right) = \left(- \frac{172 \sqrt{11}}{11} \sin{\left (\frac{\sqrt{11} t}{2} \right )} + 10 \cos{\left (\frac{\sqrt{11} t}{2} \right )}\right) e^{- \frac{3 t}{2}} \end{equation*}

Is equation 1 true with this solution set?

Eq(simplify(3*diff(xsoln,t)),simplify(ysoln))

\begin{equation*} \mathrm{True} \end{equation*}

Is equation 2 true with this solution set?

Eq(simplify(diff(ysoln,t)),simplify(4-3*ysoln-15*xsoln))

\begin{equation*} \mathrm{True} \end{equation*}

So that's it. Out answers are indeed given by

Eq(x(t),xsoln)

\begin{equation*} x{\left (t \right )} = \frac{4}{15} + \frac{313 \sqrt{11}}{165} e^{- \frac{3 t}{2}} \sin{\left (\frac{\sqrt{11} t}{2} \right )} + \frac{71}{15} e^{- \frac{3 t}{2}} \cos{\left (\frac{\sqrt{11} t}{2} \right )} \end{equation*}

and

Eq(y(t),ysoln)

\begin{equation*} y{\left (t \right )} = \left(- \frac{172 \sqrt{11}}{11} \sin{\left (\frac{\sqrt{11} t}{2} \right )} + 10 \cos{\left (\frac{\sqrt{11} t}{2} \right )}\right) e^{- \frac{3 t}{2}} \end{equation*}

Step response of a first order system with derivative input contribution Matlab

2016-02-18T04:37:00-05:00

The response of a system to a sudden excitation is often modeled as a step response. The following is an example of how to obtain the step response of a simple system. It illustrated the difference between a system with and without so-called numerator dynamics. Such dynamics represent an interaction due to velocity induced forces, such as those in viscous fluid dynamics or caused by a dashpot (also fluid-driven).

Consider the transfer function of a system given by

\begin{equation*} 4\dot{x}+x=g(t)+\dot{g}(t) \end{equation*}

\begin{equation*} T(s)=\frac{s+1}{4s+1} \end{equation*}

while for a second system given by

\begin{equation*} 4\dot{x}+x=g(t) \end{equation*}

\begin{equation*} T(s)=\frac{1}{4s+1} \end{equation*}

Find the response to an input of \(5u(t)\).

Given that the amplitude of the step input is 5, it's easiest to simply multiply the transfer function by five and use a unit step function, allowing us to use the Matlab step function.

sys1=tf([1 1]*5,[4 1])
sys2 = tf([1]*5,[4 1])
step(sys1);
hold on;
step(sys2)

sys1 =

  5 s + 5
  -------
  4 s + 1

Continuous-time transfer function.


sys2 =

     5
  -------
  4 s + 1

Continuous-time transfer function.

Alternatively, using lsim.

sys1=tf([1 1]*5,[4 1])
sys2 = tf([1]*5,[4 1])
t=0:.01:40;
u=t*0+1;
lsim(sys1,u, t);
hold on;
lsim(sys2,u,t)

sys1 =

  5 s + 5
  -------
  4 s + 1

Continuous-time transfer function.


sys2 =

     5
  -------
  4 s + 1

Continuous-time transfer function.

In both cases, the blue line represents the sys1 response, and the orange line the sys2 response. This can be demonstrated by plotting them individually.

The effect of the \(\dot{g}(t)\) term is to effectively jump start the response at a higher level, equivalent to 5/4, which are two numbers you should see in the sys1 transfer function.

Using SymPy and Jupyter to reduce 5 linear equations to a single equation in two variables.

2016-02-15T13:00:00-05:00

Solving a variant of problem 6.3, Gray, Costanzo, and Plesha (Gray, G.L., Costanzo, F., and Plesha, M.E., Engineering Mechanics: Dynamics, McGraw Hill, New York, 2010.), where kinetics are added (\(I_1\), \(I_2\), \(I_3\)).

For my students: This is Wright State University, ME 3210, Quiz 2, Spring 2016

The problem statement is: A Moment \(M\) is applied to the rightmost gear (gear 3). Find the acceleration of the gear as a resulting moment presuming inertias \(I_1\), \(I_2\), \(I_3\).

Figure awaiting copyright approval. Dummy graphic in place for the moment.

Figure 1: Gear figure from Gray, Costanzo, and Plesha.

I am using Python 3.5 in Jupyter (formerly iPython). The original notebook is available at my Github examples repository.

 # import symbolic capability to Python
 from sympy import *
 # print things all pretty
 from sympy.abc import *
 init_printing()
 # Need to define variables as symbolic for sympy to use them.
 r_A, r_B, r_C, r_D, F_AB, F_DC, M= symbols("r_A r_B r_C r_D F_AB F_DC M", real = True)
 I_1, I_2, I_3 = symbols("I_1 I_2 I_3", real = True)
 theta_1, theta_2, theta_3 = symbols("theta_1 theta_2 theta_3", real = True)

Defining equations. In SymPy, and equation is an expression that is equal to zero. All terms must be brought to the same side in defining it.

 EA = F_DC *r_D  - I_3 * diff(theta_3(t),t,t)
 EA

\begin{equation*} F_{DC} r_{D} - I_{3} \frac{d^{2}}{d t^{2}} \theta_{3}{\left (t \right )} \end{equation*}

 EB = r_B* F_AB - r_C* F_DC - I_2 * diff(theta_2(t),t,t)
 EB

\begin{equation*} F_{AB} r_{B} - F_{DC} r_{C} - I_{2} \frac{d^{2}}{d t^{2}} \theta_{2}{\left (t \right )} \end{equation*}

 EC = r_A *(-F_AB) + M - I_1 * diff(theta_1(t),t,t)
 EC

\begin{equation*} - F_{AB} r_{A} - I_{1} \frac{d^{2}}{d t^{2}} \theta_{1}{\left (t \right )} + M \end{equation*}

 E1 = diff(theta_3(t),t,t)* r_D - diff(theta_2(t),t,t) *r_C
 E1

\begin{equation*} - r_{C} \frac{d^{2}}{d t^{2}} \theta_{2}{\left (t \right )} + r_{D} \frac{d^{2}}{d t^{2}} \theta_{3}{\left (t \right )} \end{equation*}

 E2 = r_B * diff(theta_2(t),t,t) - diff(theta_1(t),t,t) *r_A
 E2

\begin{equation*} - r_{A} \frac{d^{2}}{d t^{2}} \theta_{1}{\left (t \right )} + r_{B} \frac{d^{2}}{d t^{2}} \theta_{2}{\left (t \right )} \end{equation*}

 solve(EA,diff(theta_3(t),t,t))

\begin{equation*} \left [ \frac{F_{DC} r_{D}}{I_{3}}\right ] \end{equation*}

 solve(EB,F_AB)[0]

\begin{equation*} \frac{1}{r_{B}} \left(F_{DC} r_{C} + I_{2} \frac{d^{2}}{d t^{2}} \theta_{2}{\left (t \right )}\right) \end{equation*}

I'm going to do substitutions in stages, eliminating one variable at a time for clarity.

 step1 = EC.subs(F_AB, solve(EB,F_AB)[0])
 step1

\begin{equation*} - I_{1} \frac{d^{2}}{d t^{2}} \theta_{1}{\left (t \right )} + M - \frac{r_{A}}{r_{B}} \left(F_{DC} r_{C} + I_{2} \frac{d^{2}}{d t^{2}} \theta_{2}{\left (t \right )}\right) \end{equation*}

 step2 = step1.subs(F_DC,solve(EA,F_DC)[0])
 step2

\begin{equation*} - I_{1} \frac{d^{2}}{d t^{2}} \theta_{1}{\left (t \right )} + M - \frac{r_{A}}{r_{B}} \left(I_{2} \frac{d^{2}}{d t^{2}} \theta_{2}{\left (t \right )} + \frac{I_{3} r_{C}}{r_{D}} \frac{d^{2}}{d t^{2}} \theta_{3}{\left (t \right )}\right) \end{equation*}

 step3 = step2.subs(diff(theta_1(t),t,t),solve(E2,diff(theta_1(t),t,t))[0])
 step3

\begin{equation*} - \frac{I_{1} r_{B}}{r_{A}} \frac{d^{2}}{d t^{2}} \theta_{2}{\left (t \right )} + M - \frac{r_{A}}{r_{B}} \left(I_{2} \frac{d^{2}}{d t^{2}} \theta_{2}{\left (t \right )} + \frac{I_{3} r_{C}}{r_{D}} \frac{d^{2}}{d t^{2}} \theta_{3}{\left (t \right )}\right) \end{equation*}

 step4 = step3.subs(diff(theta_2(t),t,t),solve(E1,diff(theta_2(t),t,t))[0])
 step4

\begin{equation*} - \frac{I_{1} r_{B} r_{D}}{r_{A} r_{C}} \frac{d^{2}}{d t^{2}} \theta_{3}{\left (t \right )} + M - \frac{r_{A}}{r_{B}} \left(\frac{I_{2} r_{D}}{r_{C}} \frac{d^{2}}{d t^{2}} \theta_{3}{\left (t \right )} + \frac{I_{3} r_{C}}{r_{D}} \frac{d^{2}}{d t^{2}} \theta_{3}{\left (t \right )}\right) \end{equation*}

Finally, the acceleration of the right most gear as a function of the input moment

 solve(step4,diff(theta_3(t),t,t))

\begin{equation*} \left [ \frac{M r_{A} r_{B} r_{C} r_{D}}{I_{1} r_{B}^{2} r_{D}^{2} + I_{2} r_{A}^{2} r_{D}^{2} + I_{3} r_{A}^{2} r_{C}^{2}}\right ] \end{equation*}

Performing eigenvalue analysis/reconstruction in Python

2016-02-01T12:28:00-05:00

If you are new to Python, this jumps in a bit quickly. I'd suggest looking at Numpy for Matlab Users before reading this. The following is simply a brief demonstration of using Python 3.5 (or greater) for eigenanalysis. The original notebook is available at my github examples repository. A few things to note:

If you use an earlier version of Python (than 3.5), the @ operator isn't defined "yet". To perform matrix multiplication, you need to type use numpy.dot(A,B) in place of A@B. That will require you to start with import numpy before doing any of this.
I demonstrate for a non-symmetric matrix. For a symmetric matrix, you should use eigh. The h in eigh means Hermetian, and is a more general definition than symmetric. If you don't deal with complex valued matrices, it's irrelevant to you. Using eigh leverages the properties of Hermetian matrices in the solution process resulting in potentially faster and more accurate results than the more general eig code. For a small matrix, this is irrelevant, but it becomes important for more substantial calculations.
I show a couple of tips later that matter for larger matrices (avoiding the inverse). Be aware of them. Why may or may not matter to you, but when you get to big or sensitive problems, they do.

The first thing I need to do is import a couple of tools (Scipy, and its linear algebra package).

import scipy as sp
import scipy.linalg as la

We are going to attempt to solve for

\begin{equation*} A r = r v \end{equation*}

where \(A\) is the matrix, and \(r\) represents the right eigenvectors, while \(v\) represents the eigenvalues. We are also going to obtain the left eigenvalues as well for later use.

A = sp.array([[1,2,3],[4,5,6],[7,8,9]])  # Defining the array (I'm avoiding using the matrix class)
# It's a personal preference, and I'm still not locked into it myself.
(v, l, r) = la.eig(A, left = True) # You can read the help, buy the left eigenvectors don't get created without this.
v = sp.diag(v)  # by default, eig puts the eigenvalues in a 1-D array. We will need a diagonal matrix in a moment.
print(l)
print(v)
print(r)

[[-0.46454727 -0.88290596  0.40824829]
 [-0.57079553 -0.23952042 -0.81649658]
 [-0.67704379  0.40386512  0.40824829]]
[[  1.61168440e+01+0.j   0.00000000e+00+0.j   0.00000000e+00+0.j]
 [  0.00000000e+00+0.j  -1.11684397e+00+0.j   0.00000000e+00+0.j]
 [  0.00000000e+00+0.j   0.00000000e+00+0.j  -9.75918483e-16+0.j]]
[[-0.23197069 -0.78583024  0.40824829]
 [-0.52532209 -0.08675134 -0.81649658]
 [-0.8186735   0.61232756  0.40824829]]

These should be identical based on the eigen-equation. They are to the default precision.

print(v)
print(la.inv(r)@A@r)

[[  1.61168440e+01+0.j   0.00000000e+00+0.j   0.00000000e+00+0.j]
 [  0.00000000e+00+0.j  -1.11684397e+00+0.j   0.00000000e+00+0.j]
 [  0.00000000e+00+0.j   0.00000000e+00+0.j  -9.75918483e-16+0.j]]
[[  1.61168440e+01   3.55271368e-15   1.77635684e-15]
 [ -2.49800181e-15  -1.11684397e+00  -2.77555756e-17]
 [  2.79947848e-15   4.62304004e-16   9.86076132e-32]]

In reality, one should never ever use the inverse function unless the actual answer you want is the inverse itself (which I've never seen for a real problem). What you typically want is the inverse of a matrix times another matrix or vector- which is the solution to a linear algebra problem. We can use the solve function to obtain this. This is arguably no better, but it is illustrative. For a larger problem, the benefit is easier to demonstrate.

la.solve(r,A)@r

array([[  1.61168440e+01,   3.05037570e-15,  -4.48960353e-17],
       [ -2.74605686e-15,  -1.11684397e+00,   3.95339943e-16],
       [  3.86524889e-15,  -7.16944125e-16,  -3.70074342e-17]])

This is "rebuilding" the original matrix from the eigensolution. Looks pretty good.

r@v@la.inv(r)

array([[ 1.+0.j,  2.+0.j,  3.+0.j],
       [ 4.+0.j,  5.+0.j,  6.+0.j],
       [ 7.+0.j,  8.+0.j,  9.+0.j]])

Avoiding the inverse is a bit uncomfortable in this case, but a bit of doodling yields that

\begin{equation*} A B^{-1} = \left(\left(A B^{-1}\right)^T\right)^T = \left(\left(B^{-1}\right)^T A^T \right)^T \end{equation*}

Noting that

\begin{equation*} \left(B^{-1}\right)^T = \left(B^{T}\right)^{-1} \end{equation*}

we can use

r@la.solve(r.T,v.T).T

array([[ 1.+0.j,  2.+0.j,  3.+0.j],
       [ 4.+0.j,  5.+0.j,  6.+0.j],
       [ 7.+0.j,  8.+0.j,  9.+0.j]])

For the left eigenvectors, they are actually simply the right eigenvectors of the transpose of the matrix, so

\begin{equation*} A^T l = l v \end{equation*}

Below I lazily obtain the eigenvalues using the left eigenvectors, with an inverse.

la.inv(l)@A.T@l

array([[  1.61168440e+01,   1.77635684e-15,   8.88178420e-16],
       [  1.94289029e-15,  -1.11684397e+00,  -4.99600361e-16],
       [  1.66684734e-15,   1.64791705e-16,   9.06493304e-17]])

Installing the AIAA LaTeX Package

2015-11-12T11:00:00-05:00

Every once in a while, I am asked how to install this package. I'm not the author... the package lists the authors. However, they've done a wonderful job, and enough work, so I'm going to pick up a bit of slack on their behalf with what amounts to 10 minutes of effort, and a few corrections as you send them to me.

On all platforms

I presume you've already downloaded the package and unzipped it.

The first thing you need to do is process aiaa.ins by getting to the command line on your machine and typing

latex aiaa.ins

or by loading the aiaa.ins file into your favorite LaTeX editor and compile with the LaTeX button or menu item (there are a ton of options depending on your particular installation. ). Either will work, so if you don't know where your command line prompt is, you don't have to use it.

Installation on a Mac

Then, make sure the pieces and parts are in the right places.

On a Mac, the aiaa.bst file has to be put in

~/Library/texmf/bibtex/bst

The ~/ means your home directory. To get there, if you don't know how to use Terminal:

Go to the Finder
Select Go to Folder from the Go menu.
Enter ~/Library

You can now navigate to the folder you need to get to. If it does not exist, you will need to create all of these folders in this nest so that your aiaa.bst file resides inside the bst folder nested as shown above.

The remaining entire folder (currently aiaa-3.6.1 as of this writing) that is created by unzipping aiaa.zip (or what ever archive is the package) must be put into ~/Library/texmf/tex/latex.

You might want to make a Shortcut Link by holding the command and option keys and dragging the aiaa-3.6.1 folder to the desktop. If you see an arched arrow before you unclick, you will have a Shortcut link. If not, don't let go. You are moving it and that's making things worse.

On MikTeX (most common on Windows)

See On All Platforms above and perform those steps.

Create a local texmf tree in MiKTeX per the MiKTeX website. I don't think I can write this up any better.
Make the new package known to MiKTeX: refresh the MiKTeX filename database. To do this, click "Start/ Programs/ MiKTeX 2.x/Maintenance/ Settings" (or similar) to get to the MiKTeX options, click the button "Refresh FNDB". The installation is complete.

Linux

If you have Linux, and have installed LaTeX, I am going to presume you are sufficiently capable of figuring this out via an internet search. Sorry. That's how I would have to figure it out!

To test (all platforms)

Make a copy of the aiaa-3.6.1 folder in your Documents folder. Remove the files with extensions .cls and .bst. Load the file aiaa.tex into your favorite LaTeX compiler. Try to compile it using LaTeX. If it works, great. If not, you will get a warning that LaTeX couldn't find something. Please read the logs and follow the instructions again.

A Brief Introduction to SciPy, NumPy, Matplotlib, and PyLab

2015-09-18T11:21:00-04:00

Whenever someone first begins taking any substantial steps into using Python for math/science/engineering, they face a confounding situation relating the packages SciPy, NumPy, Matplotlib, PyLab. Everything I say here is documented in detail on SciPy.org, or other places that further digging can uncover. If you want a more in depth understanding, that's the place to go. My goal here is to clarify what's going on when you look at example programs in terms of when to use what package, or more precisely, what steps you can take to determine what package/s to use for your purposes.

First, let me point out that many of the codes you will see on the web will likely have redundant calls, or will not suite your style or needs. In forums where feedback is enabled, you can often hunt for the best version. Nothing should be taken as gospel while you develop an understanding. If something works, well, it works. It may not be the best way for you in the long run, but it is a way that will work in the short term.

Instead of an accurate history of development, I'll piece together a reasonable assessment of the relative roles.

NumPy

NumPy, the Numerical Python package, forms much of the underlying numerical foundation that everything else here relies on. If you don't do a lot of sophisticated math, this might just be enough for you. The odds are against it simply by the fact you are taking time to read this blog. NumPy adds N-dimensional array capabilities and some linear algebra, Fourier analysis, and random number capabilities. It also adds some tools for connecting to compiled languages and applying functions to arrays (broadcasting) similar to Matlab®. For a quick overview, a please look at a table comparing NumPy commands to equivalent Matlab® commands. A brief summary: NumPy allows Python to do some substantial math.

import numpy as np
dir(np)             # This will provide a seemingly overwhelming
                    # list of the functions of NumPy
help(np)            # Help is always available for NumPy

Note that when we import a package in this form (import numpy as np) we are requiring our code to call functions with the lead np.. For instance

import numpy as np
np.log2(8)

returns \(3\), as \(2^3=8\). For this form, log2(3) will return an error, as log2 is unknown to Python without the np..

We could have, less advisedly, used from numpy import *, which means to bring everything in to the top level name space. The name space is simply the available named commands. To illustrate this, quit Python, rerun it, and try

from numpy import *
log2(8)

which returns \(3\). However, np.log2(3) will no longer work.

Why does Python do all of this (as compared to Matlab®)? It keeps the list of available variable names under control, something that can falter when more and more functions are defined.

Note that we can also

import numpy as np
help(np.log2)       # a function,
help(np.ma)         # or a subpackage within numpy

from numpy import ma # Import only the masked array package

The best practice when coding is to import only what you need. It enables codes to start up faster (less time loading libraries) and leaves names available for your use as variable names. Further, using import numpy as np keeps those NumPy commands separate from your workspace. An example that happens in other languages is that i = sqrt(-1) by definition. However, you can write over this with i=1, perhaps in a loop. Elsewhere, when you go to use i in an expression, it no longer carries the expected value of sqrt(-1). The more complicated the code, the more likely this becomes.

When working interactively, it is often fine to take a softer (expedient) approach and simply from np import *.

SciPy

SciPy, Scientific Python, adds substantial capabilities to NumPy. For Matlab® users, it's very much like many of the core toolboxes. If you import scipy as sp, you have also by default imported the core capabilities of NumPy, making importing NumPy almost redundant. The

import scipy as sp
dir(sp)             # This will provide a seemingly overwhelming
                    # list of the functions of NumPy
help(sp)            # Help is always available for NumPy,
help(sp.log2)       # Some NumPy functions are repeated in
                    # availability from NumPy. This should be
                    # redundant, but there seems to be some
                    # exceptions. When in doubt, try it both ways.

Be aware that if you import scipy as sp, but don't also import numpy as np, you will have to use sp.function to call function from numpy. I've recently come to the conclusion for my needs that using import numpy as np is pointless, and that simply importing SciPy's and accessing all of NumPy's capabilities from SciPy is simpler and more consistent.

Matplotlib

This is the most popular plotting (data visualization) routine package for Python. Since this is an introductory document, I won't do anything other than suggest you start here. If it doesn't fit your needs, you can use a more advanced (and possibly harder to use) package. I'm not advocating against them... unless you are new. Start here. Many users with Matlab experience will be capable of taking the step to using import matplotlib.pyplot as plt as the commands are sufficiently similar to Matlab® that the challenges are modest with Google at hand. One unfortunate oddity is that the author of Matplotlib, John Hunter, wrote a substantial amount of computational tools and embedded them within Matplotlib. This can provide a bonus, as you don't need to then import them from elsewhere. However, they are typically inferior to SciPy versions, and so best to be avoided in substantial circumstances. The debate about what to do about this overlap continues, and although I never knew Dr. Hunter, I sense that he was a great enough person to recognize that at this point they should likely be deprecated and eventually removed. That is actually a hard thing to do, as many codes already rely on these functions, and it would take a lot of re-writing/corrections/testing to extricate from everywhere, so I empathize with those working towards this.

My advice: If it's in SciPy, use it from SciPy.

PyLab

PyLab is actually embedded inside Matplotlib and provides a Matlab®-like experience for the user. It imports portions of Matplotlib and NumPy. Many examples on the web use it as a simpler Matlab®-like experience, but it is not recommended anymore as it doesn't nurture understanding of Python itself, thus leaving you in a limited environment. Of course, you aren't forced to, but it is much like a paper box. It is not a bad place to start, and can simplify your learning curve, but you may eventually decide that you were better off not using it. It is a matter of philosophy, and I'll leave that debate to others. The thing to be aware of when using PyLab is that it is importing many functions directly into the namespace. If you later decide you want to code with a cleaner namespace, you are likely to need to change your function calls from, for example, linspace to sp.linspace.

My time is running out, so my conclusion will have to wait. However, hopefully this gives you some idea of how these pieces fit together, and now you can understand a bit of how these code snippets you see on the internet behave.

Suggested Introductions

This can be an overwhelming amount of material, so I suggest skimming for awareness (and an occasional refresher), then referencing as needed. Sample codes/notebooks are certainly an easier way to get started than trying to write code from scratch.

A Brief Introduction to SciPy, NumPy, Matplotlib, and PyLab
Scientific computing with tools and workflow
The Python Tutorial through chapter 4.

Creating Good Graphics and Archiving Your Data

2015-09-14T11:21:00-04:00

Updated March 11, 2018.

Introduction

You've spent days/months/years collecting/analyzing your research topic. You put the requested information into your paper/presentation, turn it in, and viola, you're done, right? Not really. What you've just generated is an electronic document of which the only remaining part may be a hard copy in a few years. Further, without all of the splendid data that you used to generate your charts they are now most likely uncorrectable.

This is the case for a large portion of the theses and project reports that students and professionals develop. Many times the only result of a significant amount of effort is a digital photograph of a specific representation of the data. Changing the representation of the data in the future isn't a simple task unless a few simple steps were taken to assure their permanency. Further, setting your graphics in the wrong format significantly degrades the presentation of the effort on paper, on screen, or both. This short document illustrates simple steps that must be taken in order to archive your work for future access.

Data storage

Plotting your data and saving the plot is not archiving the data. It is converting it to a visual representation, one that you chose. It may not be the one that you, or the people you send it to, want in the future. Archiving means saving the raw data. Every amount of data processing that moves away from raw data destroys information that is almost always irreversible. Calculated quantities such as mean, variance, integrals, etc., do not retain all of the information in the raw data. Further, saving only those quantities raises an error in their calculation into a monumental blunder should their calculation have somehow been faulty. It is important to archive your data in the least processed form possible. If the processing is very time consuming, archiving the processed result is also a good idea. However, rarely is the cost of archiving raw data excessive.

How to store your data

Data should always be stored in raw ASCII (American Standard Code for Information Interchange), or raw text, files. Other formats can additionally be used. However, those formats may become unreadable in 10-20 years. The ASCII format has been around for a long time, and is the format of the programming languages used to write the codes that read other formats, so it's a good bet that this format will exist indefinitely.

Data should be stored in column delimited format. You can use whatever character you like for delimiting. Spaces, commas, tabs, all can work fine, but make sure that none of those characters are part of your data. Almost all computer codes can save to an ASCII file of some type. This file will almost always be larger than when using other formats. Further, your program may complain that you will lose information in saving in this format. Often, this is useless information. However, try to find out what it is. In many cases you should also save in native format for the code. In fact, you should almost always save in both formats. The ASCII format is your emergency backup for when you can no longer read a data file written by a code that has either a) been extinct for many years, or b) is unavailable to you due to licenses at your new place of employment. [1] If there is a standard format for your field, you should also save in that format. This means up to three archived data files in the place of one. If this seems like a lot, then you don't think much of your data, and this document is irrelevant.

Documenting your data

Just having your data stored in files is not enough. It must be documented. This documentation may be in your report/thesis. If so, this document must be part of the data archive. Further, this documentation must be provided in ASCII format too! We really don't know if a MS Word '95 file will be readable in the year 2020. Probably not. If that's the case, then when you or another researcher come back to this data in the future to see if some new processing technique yields new and exciting insight, you'd better hope that the data is documented in an ASCII (text) file. Often, this file is named README.TXT. The reason should be readily apparent. This is the first document you should look at.

A good form to use is either the reST or Markdown standards. The advantages of these are they are ASCII, but very readable. I have more experience with reST and favor it's ability to be re-rendered in a slew of other formats using tools such as Sphinx. In fact, the source of this very page. Further, while perusing my website (not this blog) you may notice that there is a link to the left that almost always allows you to see the original source in reST format.

Sets of data described by this document should be in the same directory as this document, or subdirectories beneath it. This document must contain:

A description of the experiment, including equipment used (down to serial numbers!), names of files corresponding to pictures useful for recreating the experiment, and enough information to find final published documents using this data (the actual citation is preferred). If the data is the output of a code, a version number and the input information should be supplied. If at all possible, the code itself should be supplied, as well as the compiler requirements (e.g. version 4.0 of Matlab, or LaTeX, or MS Excel 95)
Nomenclature for directories/subdirectories and files so that the circumstances (case information) of each file can be ascertained.
Your name and where you think you are going. Some way to contact you.
Names of codes that can read this data (for each file type). This can also be complex. As much as possible, presume the reader has no clue what to do to use this data. Most likely, you will be the reader in the future.

Archiving your results: File Formats

The results of your collected data are the computational analysis and resulting plots and figures resulting from them. No doubt this took a great deal of effort. If all of it was generated manually, then you have made life hard on yourself. Most tasks involving data processing plotting are easily automated with modern tools. In performing many tasks by hand, you require the future user of the data to know each and every step you followed to process then plot the data. Guess what: this is the second part of the README.TXT file. Most likely, a user will want to change one or two simple steps of the process. The more complex the process, the more specific the instructions will be.

Plotting

Languages such as Octave, gnuplot, MATLAB, Mathematica, IDL, Python matplotlib etc. allow scripting. A compiled language can also be used with the same results. You should provide a file that when executed will generate the files that became your plots. If the data processing is extensive, the processing and plotting (or rendering) should be separated into two or more tasks, with two or more scripts. Ideally these scripts should not prompt the user for information, but define answers to pertinent questions as variables that can be edited. Better yet is to have one script for each data file with the names easily identifiable. For example: plotfrfdata.m and frfdata.txt.

These files need not read directly from the raw ASCII date file but should make clear how to load data from the text file if necessary.

File formats for graphics

Use of the wrong graphics formats for data presentation is the best way to make good results look sloppy. The following is an incomplete list of graphics file formats. It lists the most reliable ones for the tasks stated. Reliable means commonly acceptable and readable.

Photographs

TIFF (Tagged Image File Format) is a very, very (usually) high quality format for graphics such as scanned images and photographs. It also results in some very large files. Joint Photographic Experts Group (JPEG or JPG) allows significant compression with minimal loss of the same types of data. However the compression can lose some information and thus jpeg shouldn’t be used when image processing might later be performed. Both work well when various shades of color/brightness are used, with relatively soft edges. Storing a photograph in any other format likely makes no sense at all. Using these formats for graphics other than scanned images or photographs is not recommended. A very coarse description of these formats is that for each pixel they record the appropriate color and darkness to plot. JPG uses schemes to compress this data. In the compression process, JPG (before 2000, which is far form standard) uses techniques which cause strange aberrations in data with lines or text.

Preferred archive format; TIFF, JPG (high quality)
Preferred presentation format: JPG

Line art, plots, graphs, computer renderings of objects with computer drawn axes/lines, text

Vector graphic formats are ideal for these cases. Encapsulated PostScript (EPS) or Scalable Vector Graphics (SVG) is the best way to go, with Portable Document Format (PDF) a close second. Windows Meta File (WMF) may work if you are confident that you are getting vector graphics. Encapsulated PostScript (EPS) or Portable Document Format (PDF) done correctly doesn't know what a pixel is. Thus it prints to infinite resolution by telling the output device simply that there is a line between two locations (not pixels) and that the output device should adjust pixels in its resolution to make it as perfect as possible. This differs from JPG/TIFF which can alias and look quite bad as they are viewed closer or blown up in size. EPS and PDF can be scaled up or down without loss in quality. Generating a JPG or Graphics Interchange Format (GIF) and converting it to EPS is not equivalent and not recommended. All of the vector information is destroyed in generating the JPG/TIFF/PNG(Portable Network Graphic)/GIF and cannot be recovered. Screen captures currently also result in poor quality graphics. There are some exceptions. If you think you have the exception, print your graphic at 100 times the capture size. If the text looks jagged to your eye at any level on a 300 dpi printer, you don't have a vector graphic.

For example, observe Figures 1 and 2. They are sections of complex graphics. Figure 1 was saved as a JPG, then imported and edited in an unknown package. Figure 2 is the one I regenerated. Note the vastly improved cleanliness of the line. By saving as a vector graphic, the line in Figure 2 is now editable by hand or using an editor such as Adobe Illustrator. Figure 1 cannot easily be edited to improve the appearance of the line.

How to generate vector graphics.
Language-platform	Command	Help
Matlab	print -depsc2 filename	help print
Python matplotlib	savefig with the ext="svg" option. Alternatively eps or pdf.	help(plt.savefig)
Mathematica	Export["filename.eps","eps"]	Help menu
MS Windows	Install a PS driver, print with this driver. This won't work with all programs in Windows.	Google it. Sorry. Edit text file to start with %!
Macintosh	Print, note options at bottom left of dialog box	PDF is built-in

Figure 1: Zoomed line from a JPG file. Zoom in a factor of 10 times and notice the jaggedness of the line.

eps zoomed, svg format

Figure 2: Zoomed line from an EPS file. Zoom in as much as you want. The line will be no more jagged than the resolution of your screen.

The software Inkscape can convert to vector graphics, but it's much better to get it right the first time as the results can still show undesirable effects due the the need for potrace to make judgement calls during conversion.

Drawings

The PNG and GIF formats are ideal for graphics with swaths of uniform colors. PNG is preferred by some because there are no license restrictions on the format, but GIF is more common. However, the licenses and concerns that some had for the GIF format ended with the expiration of all patents on the technology in August, 2006. Either of these do well, resulting in high quality graphics with small file sizes. The problem with them is that lines and text can also alias. In most cases it is best to simply use EPS in these cases as well. EPS stores the drawing as a set of objects that can be edited as an object instead of as a painting. PNG and GIF do not understand objects so much as pixels.

Combined

When you have text on a photograph, what do you do? Use EPS again (or PDF). The JPG or TIFF is embedded in the file as an object, and the text is embedded on top of it. With these formats you can combine the best of both worlds.

Archiving Data: Storage

Archiving is more than how the data is organized. It also included physically where the data (the actual storage media) is put and under what conditions it is stored. Choice of media is of critical importance. All of the preceding advice is eventually worthless if the media on which you store your information fails. In my observation most computer users who are prepared for media failure have suffered from failure before. Those who haven't often have too much faith that their hard drive/tapes/floppies/CDs will last forever.

Archiving Media

Magnetic-based media such as floppy disks and hard drives have a half life of between five and seven years depending on storage conditions. Technically, that means the strength of the magnetic fields drop by 50% in that time. Practically, it means there is a significant chance of data loss in that time. Optical media tends to be more robust, but varies depending on multiple factors.

Floppy disks tend to fair the worst because they are typically manufactured as a commodity item where quality is of minimal concern. This wasn't always the case, but it certainly is now. Floppy disk drives are also manufactured cheaply and their read/write quality varies. Any data currently on floppy disks (zip disks included) should be expected to last no more than a couple of years reliably (if the data is critical, this is insufficient). My advice is to either throw out the floppy disks, or copy the information to a better medium, then throw out the disks. My experience with old floppy disks is mixed.

Hard drives tend to fair better, being mostly reliable for the four or five year life of the typical computer. However, storing something on a hard drive on your computer is not archiving. You can archive to an external hard drive, but it is critical that the hard drive not be accessible again except in an emergency. It cannot remain accessible electronically to any computer. If it does, it is vulnerable to viruses, power surges, user errors, etc. Hard drives do not provide sufficient long term reliability given that their cost exceeds more reliable methods.

Tapes tend to be the most reliable of magnetic media. Life expectancies of 10-30 years are common, but only under ideal conditions. The National Bureau of Standards publication, Care and Handling of Computer Magnetic Storage Media, recommends that magnetic tape be stored at 65 +/- 3 degrees Fahrenheit and 40% +/- 5% Relative Humidity. For whatever reason, I’ve observed that tapes are rarely used by individuals for archiving. This was perhaps erroneous until the advent of optical media.

Optical media can be much more reliable, but it is hard to produce, and the cheaper media is not more reliable than magnetic media. CDs/DVDs can fade over time. Life of the media depends on the sealing method, reflective layer, organic dye used, and user storage conditions. Re-writable media should no be used for archiving as they are specifically designed to be changeable after the first burn, i.e. the “burn is not permanent”. Thus they are also more vulnerable to decay over time and more susceptible to damage. Further, different formats, e.g. CD-R, CD+R, DVD-R and DVD+R use unique algorithms/concepts for tracking data on the media and providing redundancy. DVD+R is the most robust format, coming after all of the others, and being designed to overcome the shortcomings of the others. McFarland has a rather long and detailed document on this, and recommends using only Taiyo Yuden DVD+R at the current time, available from the SuperMediaStore.com, purported to be the only online US distributor that guarantees that their media is certified Taiyo Yuden. The Taiyo Yuden FAQ also contains useful information. Although this citation is a blog, given the depth of discussion and responses, I believe it warrants respect. Unfortunately I haven't looked for alternative citations on this topic. However, my recollection of articles during the development of DVD writable media agree with the substantial part of this document.

Usage of Graphics in Documents

Shortly after reading this document, you will find out that MS PowerPoint will not accept EPS files for show. They may print. I don't know. However, since that’s not the intended use of MS PowerPoint, it doesn't matter. My answer to that is that this document covers archiving, not usage. Although there is overlap, and the information and explanations within still apply, practical limitations always take precedence. For the sake of usage in MS PowerPoint, I convert my EPS files to PNG or GIF format. In the future, JPG2000 format may be a better choice, but compatibility is a major issue when generating a presentation.

Care of Media, Transcription, and Redundancy

Media has a finite storage life. Proper care of the media can extend that life. I currently don't have the answers for storage of all forms of media, and am unlikely to ever have them. It behooves the reader to determine what the optimal storage environment is for their media. As most media is designed for use in a typical office environment, it isn’t a great leap to presume that room temperature with approximately 40-60% humidity is likely near optimal. This is in fact the case for the media that I’ve researched (tapes). All storage media should also be kept in the dark (light is energy, can be converted to heat, which causes diffusion) and in as clean an environment as possible. Even for media that can be cleaned, dirt can not cause more good than harm. It can eventually end up in places it shouldn’t me.

Further, magnetic media by its very nature should not be exposed to magnetic fields. Fortunately magnetic fields drop rapidly away from the source so keeping the media at least 2” away from a magnetic field in a typical office should be sufficient. It is typically easy to guarantee multiple feet, making any nearby field irrelevant.

Often the technology that writes to and reads from the media is more finite than the storage life of the media. The 5\(\frac{1}{4}\)” and 3” floppy combined had a technology life of approximately 20 years total before being replaced by newer formats with higher storage densities. Many older scientists and engineers have boxes of programs on punch cards that can no longer be read. [2] As new technology becomes available, it is prudent to occasionally, say every 10 years, create an additional archive of the data in the new format (transcription). It is further prudent to understand whether that format is useful for archiving or not (long life, inexpensive, sufficiently popular/simple that there will be readers in the future, sufficiently robust to errors). The original archive should not be discarded. The benefit of discarding it (minimal) is likely heavily outweighed by the cost necessary to verify that the new media is a perfect duplicate and will also survive equally as long.

Further, as archived data is important, so is its physical protection. Buildings burn down, are flooded, robbed, boxes are lost and accidents happen. It is prudent to have a second archive in another physical location. Rarely will this ever be needed. However, if the data is important, it is certainly worth the redundancy. Most people can use their home and office for such redundant locations. I've had to use secondary archives due to failed magnetic media in the past. When one loses data it can be very trying. The cloud is a sound option for many. However, legal privacy issues such as FERPA, ITAR, and HIPPA have to be considered before using such. This is not an complete list. You are responsible to know when you are handling sensitive information.

While this document is intended to give some insight as to how to archive data, it is not intended to cover backups, which are somewhat like short-term archives that can be replaced. It is left to the reader to investigate the importance of backing up. As a quick summary, weekly backups to an external drive, or better yet, an online backup system (automatically off-site), is prudent. Depending on the rate of your work, quarterly or so archives are also prudent. Just as in archiving, the average user tends to lack sufficient paranoia.

Summary

It's your data, and your graphics. I haven't found anyone yet who has disagreed with this advice, [3] and most have immediately told stories of countless hours wasted compensating for a lack of proper archiving practice. Data must be stored and documented in a meaningful way that is absolutely clear. Regarding graphics, use EPS or PDF for all graphics that are not a pure photographs. Converting a photograph to a JPEG or TIFF is a simple task on a Mac using the Preview application. On other platforms there is free software available, or Adobe Acrobat can be used. This document is focused more on "what" than "how." Modern internet search engines are more than adequate for finding out how to perform these tasks, and they will evolve as tools evolve. While this document is far from exhaustive, it will set you on the right path to protecting your hard work for future use.

Bibliography

[1]	Using GNU software protects you from much of this by being free for use in any environment.

[2]	Ironically, it is only recently that technology is developing electronic storage media with lives longer than cards!

[3]	If you do, please help me improve it!

Using Python to find absent members

2015-06-10T11:21:00-04:00

Recently as part of my professional service activities, I was tasked to ask a substantial number of people to register in an online database with relevant information. Getting people to do such things can tack some coaxing to reach completion. That's well known. However, the time consuming part of the task is matching the lists to see who hasn't completed registering. This can be a substantial task as I would have to manually search for each of the 70 members in what is a rather long list (in the thousands). Each search isn't that long, but nobody in my team wanted to do all of the searches. It seemed rather barbaric that it couldn't be automated. So, after a bit of work, I found that it took precisely 4 lines of code in Python.

# findmissing.py
import pandas
list1 = pandas.read_csv('list1.csv') # my member database. I care
             # that they have registered on list2
list2 = pandas.read_csv('list2.csv') # list of everyone who has
             # registered. This includes people not in my member
             # database. I don't need those people in my output
delinquent = list1[~list1['Email'].isin(list2['Email'])] # all data
             # of people in my member list who have not signed up
delinquent(np.array(['Phone', 'Last Name', 'First Name'])) # print
             # the useful information

As you can anticipate, a lot is happening in a very small space. However, it's pretty easy if you parse is step by step. Admittedly, the last line would be simpler if I did it in multiple steps. On the other hand, at least I didn't execute all of this in one line of code.

Step 0

I simply put the name of my file inside the file so have it. Take my block and make a text file with the name findmissing.py. You can use another name, but you need to adapt my later instructions.

Step 1

We start with the line:

import pandas

This imports pandas, the Python Data Analysis Library. This enables us to import datafiles in comma separated value format very easily, as illustrated in the next two lines.

Step 2

list1 = pandas.read_csv('list1.csv')

imports the list of members of my group. I want to know who in this list is not in the second list. Further, I'd like to get all of the information about them. I could pull only partial information. I'll explain that in Step 5.

Note that I either have to execute findmissing.py in the same directory (folder) as the files list1.csv and list2.csv, or I need to insert paths to these files. On my computer, the complete path (including the filename) is /Users/jslater/Downloads/list1.csv. You either need to figure this out, or just run the script in the same folder.

Don't forget, you either have to name your file list1.csv or modify the code with your filename.

Step 3

I need to get the database system everyone is registering on to export all registrants to a CSV file. This is on their end to put their data in a file for you. Hopefully they have a method for you to do it on your own. If I can filter the before the export, without losing any of my people, that's fine. However, it's not necessary.

Once I do that, I can import the registrants database with

list2 = pandas.read_csv('list2.csv')

Note that we have to assure filename and path are correct/accommodated.

Step 4

We have one line left, and that's where all the work is done. Let's do this in steps, peeling from the inside out.

list2['Email']

One of the columns is named Email. You have to adjust for what you think is an appropriate identifier, whether it's a username, social security number, phone number, etc. It needs to satisfy 3 requirements: be reliably correct, be unique, and never be absent.

The result of this snippet is to create a list of all email addresses (my chosen identifier) from my registrants list. This lists a bunch of people who I don't need to know about.

isin(list2['Email'])

This is an operator that operates on the stuff passed to it (the stuff before the period). It's receiving a list and answering, for each member in the list, whether or not that member is in list1['email']. So,

list1['Email'].isin(list2['Email'])

will give me True for each member of list1 who is in list 2, and False otherwise. The result is another list, corresponding one-to-one with the members of list1. Now recall that we wanted to find out who is not registered. This is where the ~ comes in to play by inverting the results of isin.

~list1['Email'].isin(list2['Email'])

will give me a list that has False for each member of list1 who is in list 2, and True otherwise. So, we now have a list that states when it is true that a member hasn't signed up.

Step 5

Now all I need to do is pull out of list1 the information for those people who are not registered (the result of the prior step).

list1[~list1['Email'].isin(list2['Email'])]

For the sake of convenience, if I want to do anything more with this, I should probably assign it to a variable, such as

delinquent = list1[~list1['Email'].isin(list2['Email'])]
delinquent(np.array(['Phone', 'Last Name', 'First Name']))

which will list phone numbers for those who haven't registered. Phone needs to be the first line header for the phone numbers, along with Last Name and First Name.

If this helps you, or if it can be improved, please send me a note.

Why Matlab Users Should be Moving to Python

2015-06-08T23:21:00-04:00

Back in 1989, when I first entered graduate school as an engineering student, I knew a handful of languages to some modest degree. For the sake of computations, I knew Basic and Fortran 77 (the horror!). During that first year I was assigned a project that required substantial programming (for my skill set at the time). This project was almost complete with about an hour to go before the deadline. Unfortunately, this was the days of floppy drives (it was not a personally owned PC). I accidentally stepped on the switch on a power strip and damaged the files beyond my capability to recover them.

With about an hour to go a fellow graduate student, Jeff Umland- now of JPL, suggested that perhaps I should heed his advice and use Matlab. [1] The project would be simple in Matlab according to him. There was certainly no way I could complete the project in Fortran 77 in the remaining time. So, with him responding to my shouted out questions (learning under fire - ''How do you write a loop?'') I started using Matlab and finished with time to spare. That was the last time I started a project using Fortran. Fortran has its place (although it seems to be slowly fading away), but it has been clear to me ever since that for most problems facing me, and most engineers, time of development far exceeds time of program execution. When it doesn't, your prototype in a scriptable language is still not a bad first start. So, the speed cost of using an interpreted language was well worth the savings in development time.

As some might guess, I later switched to Octave. Ironically, what I liked the best about Octave initially was that it wasn't a Matlab clone, but nominally equivalent for most tasks. As time has progressed the author, John Eaton, has acquiesced to public pleading and moved further and further towards Matlab compatibility. [2] While I respect why he made the move, for my interests it downgraded Octave, although I'm likely a minority, but it is likely the right decision for the Octave ecosystem.

Incidentally, and I made this blog in Python, using Pelican, writing the source files in ReST.

[1]	I have always been a bit stubborn in my opinion. I think I'd made some positive strides as I've grown older.

[2]	Ironically, early on he observed clear reluctance on my part to convert to the Octave standards of Octave's help system compared to Matlab. I think he missed the moment when I came to agree with him about that and other things! I'm sorry!

Inside the Ivory Tower

George Floyd

Dear College of Engineering students, staff, and faculty,

Curve fitting experimental data

Fitting a line

Other forms

Power function

Exponential function

Strategies for Reducing Cheating Amongst Students

Homework

Homework Solutions

Projects/Papers/Reports

Computer Programs

Quizzes & Exams

Broad categories of cheating and strategies for preventing them are:

Inappropriately transporting information into an exam

Communicating with others

Cheating off other students' exam papers

Take-home exams

Summary

Why I Moved to Python From Proprietary Solutions and Will Not Go Back

kwargs, args, **kwargs, and *args

They don’t exist

If they don’t exist, why do I see them all over the place?

What is packing?

How does *args work in a function?

What is an example where this could be helpful?

So what about **kwargs?

Conclusion

Using Laplace Transforms to Solve a Linear Differential Equation in SymPy

Installing Python for Scientists and Engineers (and maybe Mathematicians too)

What is everything, and why you don't want to manually install it

Any platform (Mac/Windows/Linux)

Windows

Mac

Linux

Government labs

Questions?

Scientific Python Understandings: Things I want to remember

People

Pages

A Quick Introduction to Sympy

Getting cURL to work in Cygwin

Solving multiple linear ordinary differential equations in SymPy

Step response of a first order system with derivative input contribution Matlab

Using SymPy and Jupyter to reduce 5 linear equations to a single equation in two variables.

Performing eigenvalue analysis/reconstruction in Python

Installing the AIAA LaTeX Package

On all platforms

Installation on a Mac

On MikTeX (most common on Windows)

Linux

To test (all platforms)

A Brief Introduction to SciPy, NumPy, Matplotlib, and PyLab

NumPy

SciPy

Matplotlib

PyLab

Suggested Introductions

Creating Good Graphics and Archiving Your Data

Introduction

Data storage

How to store your data

Documenting your data

Archiving your results: File Formats

Plotting

File formats for graphics

Photographs

Line art, plots, graphs, computer renderings of objects with computer drawn axes/lines, text

Drawings

Combined

Archiving Data: Storage

Archiving Media

Usage of Graphics in Documents

Care of Media, Transcription, and Redundancy

Summary

Bibliography

Using Python to find absent members

Step 0

Step 1

How does `*args` work in a function?

So what about `**kwargs`?