NIIMBL Funded Research

by Mary Martialay on January 19, 2019

The National Institute for Innovation in Manufacturing Biopharmaceuticals (NIIMBL) has awarded Rensselaer a workforce development grant as part of a nearly $13 million round of recently announced funding, a value which includes cost sharing from participating organizations. The grant, in collaboration with Celegene and LumaCyte, LLC will, develop “characterization of cell therapy products by flow and laser force cytometry.” 

NIIMBL, a public-private partnership to advance innovation in biopharmaceutical manufacturing, is part of Manufacturing USA®, a network of 14 manufacturing institutes across the country that brings together industry, academia, and the public sector to propel promising research developments, accelerate new products to market, and train tomorrow’s workforce in order to secure America’s future.

NIIMBL is funded through a cooperative agreement with the National Institute of Standards and Technology (NIST) in the U.S. Department of Commerce, and leverages additional support from industry, academic institutions, nonprofit organizations, and select state governments.

A New Approach

by Mary Martialay on January 18, 2019

Rensselaer Polytechnic Institute sits atop a hill overlooking Troy, NY. A massive granite staircase connects the campus and the riverside city below. It is known as The Approach.

Initially completed in 1907, The Approach has served as a literal entry point to the university for more than a century. Its current splendor is the product of a significant renovation supported by the Louis and Hortense Rubin Foundation in the 1990s.

Since launching in 2009, The Approach Blog has chronicled the innovative ideas, processes, and endeavors of students and faculty at Rensselaer. It is about to undergo a renovation of its own.

Going forward, The Approach Blog will serve as a space to acknowledge and celebrate the multifaceted research and accomplishments of faculty and students at Rensselaer.

The posts may range in length, depth, and subject matter, but they will maintain a consistent spirit. Like the actual Approach, this blog will elevate the members of the Rensselaer community and bring them to a common area that encourages further education, collaboration, and discovery.

World of Plankton

by Mary Martialay on December 12, 2018

(Please enjoy this guest post, written by Kathleen Ruiz, Rensselaer associate professor of Integrated Arts, on the “World of Plankton” interactive touch-pool exhibit, now on exhibit at an aquarium in Burlington, VT.)

What if there were mysterious creatures on earth responsible for all the freshwater on the planet?

The World of Plankton is an interactive ecological art game installation currently on exhibit at the ECHO, Leahy Center for Lake Champlain, running until Jan 6, 2019 that takes visitors into the hidden realm of these microscopic freshwater organisms. The installation has a virtual touch-pool aquarium with original art, animation, and composed music that allows up to four players to gather around a giant digital touch table to capture and explore zooplankton, phytoplankton and fish species. Also included are interactive plankton sculptures. The entire work is surrounded by a 32’ mural.

As a media artist who works with game simulation, virtual/mixed realities, photography, and sculpture, my long-running body of work highlights the importance and complexity of freshwater ecosystems, using art and science to build critical awareness about the environment. My scholarly and artistic research is centered on simulation, perspective and empathy. In my customary style, these works are manifested both digitally and in conjunction with physical interfaces and objects, and involve artistic and scientific “real world” fieldwork experiences.

Being the originator, lead artist, designer, director and producer of the World of Plankton, I have had the honor of creating and working with an amazing team of students and alumni in the arts and game design programs at Rensselaer. We are developing original game design, two-and three-dimensional art, animation, music, sound design, artificial intelligence, and programming. Together we are committed to demystifying, engaging, educating and activating people about the importance of plankton.  Especially in our current times where we see the annihilation of environmental protections and funding, we are “keepers” of the emerging environmental knowledge, giving tiny plankton a voice and a presence through virtual embodiment. Using the virtual is a radical act of preservation and transmission of this vital knowledge as we help build a critical awareness.

We are resonating creative artistic inquiry and practice with the evolving research of the Jefferson Project, a partnership between Rensselaer Polytechnic InstituteIBM, and The Fund for Lake George in revolutionary environmental monitoring and remediation that combines a network of sensors in and around Lake George in measuring physical, chemical, and biological parameters.

The World of Plankton seeks ways to expand this inquiry and connect into a broader cultural understanding by creating artworks and environments that will enable us to see and hear what the lake is telling us as a total entity, and lead us to a deeper wisdom about how and why we need to protect our waterways. Together we are asking the questions that no one discipline alone can ask.

What are Plankton?

“Plankton” may not be a household word, but these little-known beings are vital to all life on earth. What are they? Plankton are communities of algae (phytoplankton) and animals (zooplankton) that are suspended in the water and travel with the water currents. Phytoplankton give off oxygen when they use sunlight to turn carbon dioxide into sugars. Roughly half of the oxygen in our atmosphere was made by phytoplankton. Phytoplankton also form the base of aquatic food webs. All life in the ocean and in bodies of freshwater ultimately depends on algae for food. Phytoplankton provide a rich supply of food for zooplankton and other creatures that eat them in turn, such as larger zooplankton and fish. Small fish then, are eaten by birds and larger fish. Everyone ultimately depends on plankton to survive. However, if this delicate balance of phytoplankton and zooplankton is disrupted by nutrient pollutants such as chemical fertilizers or improper septic waste, nitrogen and phosphorus will abound. These very high nutrient conditions give rise to an overabundance of algae, resulting in algae blooms, which can create toxic conditions for fish and zooplankton. Road salts as well can make intolerant conditions that will upend the delicate balance of phytoplankton and zooplankton. When we learned about this through the research of our science colleagues on the Jefferson Project at the DFWI (Darren Freshwater Institute), we realized that there is a fragile delicacy to nature that must be deeply respected. Our World of Plankton project engages users to learn about long-term effects of nutrient pollution on lakes. Informing and educating the public about the importance of valuing and preserving fresh water is the key to enduring watershed protection.   We feel it is these personal, hands-on experiences that help people understand the causes of declines in lake health and how modifications of their personal behavior can help reverse human impacts on the watershed.

We also realized that our work is a whole world endeavor as harmful algal blooms are occurring at a greater rate globally. Our World of Plankton team consists of people from many areas of the United States, China, the Caribbean, Europe, Central and South America. We all have seen declining conditions in freshwater resources firsthand in our perspective countries.

The World of Plankton submerges players in a virtual touch-pool aquarium, offering a hands-on exploration of the importance of fresh-water ecosystems. My idea was to shrink the human game players to a size so small that we could almost snorkel with the plankton and get to know them more personally. What are these tiny sea creatures trying to tell us? What is each body of water as a whole trying to tell us?

Can a game at the juncture of Art, Science and Education make a difference?

At the confluence of game design, art, and science, the World of Plankton touch pool installation includes an interactive virtual aquarium that helps to explain the role of plankton in freshwater ecology. The World of Plankton exhibit is comprised of a large touchscreen computer table that allows up to four people to view a lake food web—selecting animated algae, zooplankton, and fish—and learn about the role that each species plays in the lake ecosystem. The user can navigate at multiple scales, from microscopic algae to macroscopic fish. The exhibit also includes 3D printed sculptures of algae and zooplankton in resin (4-6” in size) so that visitors can see the magnificent detail of these fascinating organisms.  Visitors can bring the plankton sculptures to life on their mobile devices by scanning QR codes found in the sculptures. This activates original animation and composed music enabling both an aesthetic experience as well as stealth learning about their biology. To create further immersion, the entire installation is surrounded by an original 32-foot digitally painted mural on canvas that inverts the size of fish and plankton. You essentially see, hear and touch plankton in new ways through art, music and sculpture, and learn about them all at the same time. We feel this is a potent way to become more informed and activated about the very urgent issues of environmental ecology.

In its first month, the exhibit has received tremendous interest from the public. Especially exciting for me was to learn that, not only are 5 year olds seamlessly learning and interacting with the work, but the added layer of more in-depth information I designed into the project is being utilized by University of Vermont faculty and student researchers before they go out on Lake Champlain for their fieldwork expeditions.

Although this work was originally conceived as a 4-month exhibit, public interest has been so positive that we have been asked by ECHO to convert the temporary exhibit into a permanent exhibit that would inspire and educate the ECHO’s more than 150,000 annual visitors! It is through the dedication of our Rensselaer team of students, alumni and faculty artists, programmers, and scientists that we were able to achieve this goal.


We are currently developing a second game for the public, known as The Aquatic Messenger. Using virtual-reality goggles, this work takes users back in time where they are immersed in a three-dimensional, underwater world of lakes in our region. Users can explore a wide span of time, from the Cambrian Epoch 490 million years ago when the sea covered the lake, to the receding glaciers 10,000 years ago, through the arrival of Native Americans, the arrival of European colonists, the present day, and what the future could be in 2100. When completed, this game will allow users to experience historic human impacts, including the impacts of logging watersheds in the 1700s and the gradual increase in algal growth over the past three centuries due to increased nutrient inputs. This work will offer a long temporal view of how humans have been contributing nutrient pollution. We are also beginning a new mixed reality game-art simulation focused on eco-resiliency and the dynamics of harmful algal blooms (HABs).

Yes we are:

As a co-founder of the GSAS Game Simulation Arts & Sciences program in the School of Humanities, Arts and Social Sciences, this project is the kind of work I envisioned we could do at Rensselaer where transdisciplinary thinking is not only permitted, but encouraged. What we are learning is that all of us working on the project, (whether we are from game design, the visual arts, music, computer science, biology, environmental science, micropaleontology, etc.), are looking at the same thing, but from different points of view and when we come together, we ask questions and propose solutions in ways that none of us could do alone. For instance, we have realized that invasive species from other countries come here, while other countries face invasive species coming from the United States. We are not outside of nature, but each a part of it globally. As we face the climate realities of today, it is critical to come together in profound new ways to understand each other in non-colonizing ways. Yes, we can give voice to tiny, unseen plankton and learn about their plight through game simulation. Yes, we can use this technology creatively to empathize and discover other people, different cultures, and huge ranges of complexities that will enable us to think in terms of openings of systems, contingencies, and new potentials of expression.


We gratefully acknowledge the on-going support of the  Jefferson Project, the support of the NVIDIA Corporation’s Academic Research Grant that enabled the GPU used for this research and 1st Playable Productions for supporting development. Support is also provided by a grant from NYSCA (The New York State Council on the Arts) in partnership with Wave Farm: Media Arts Assistance Fund (MAAF)

[The Jefferson Project at Lake George is conducting ongoing research into how human activities may be affecting the lake. This guest blog by Kayla Coldsnow, a graduate student in the lab of Jefferson Project Director Rick Relyea, summarizes recent research published in the journal Environmental Toxicology and Chemistry. The Jefferson Project is a collaboration between Rensselaer, IBM Research, and The FUND for Lake George, founded to develop a new model for technologically enabled environmental monitoring and prediction to understand and protect the Lake George ecosystem and freshwater ecosystems around the world.]

 What did you want to know?

Applications of road salt are increasing concentrations of salt in freshwater ecosystems that we use for drinking water, recreation, industry, and agriculture. It is important that we understand how plants and animals are affected by increased salt, particularly for invasive species such as Asian clams that originally evolved in environments that can range widely in saltiness. If Asian clams can survive in aquatic ecosystems with high salt concentrations, they may be able to outcompete native clams in ecosystems impacted by road salts. Although most road salt applications contain sodium chloride, some applicators use alternatives including magnesium chloride or calcium chloride As a result, we were interested in documenting the tolerance to all three road salts.

How did you go about it?

We completed two lab experiments in which we exposed Asian clams to various concentrations of each road salt for four or eight days. At the end of each experiment, we determined how many clams were alive in each salt concentration. After the experiments, we calculated the lethal concentrations that cause 50% mortality – known as LC50 values –– for each salt and time point. LC50 values are useful in comparing the relative toxicity of different pollutants.

What did you learn?

We discovered that Asian clams are extremely tolerant to all three road salts. For reference, the typical salt concentration of freshwater (including Lake George) is 0-20 mg chloride per liter (Cl/L), while seawater is approximately 20,000 mg Cl/L. In comparison, the LC50 values for magnesium chloride and calcium chloride were approximately 1,800 and 3,600 mg Cl/L across the two experiments. For sodium chloride, the LC50 value (22,000 mg Cl/L) was higher than seawater in the 4-day experiment and still very high (10,000 mg Cl/L) in the 8-day experiment. Such strikingly high tolerance to all three salts may reflect the fact the Asian clams come from environments in Asia that range from fresh water to brackish water. This high tolerance to salt should allow Asian clams to invade and survive in a wide range of habitats around the world, including lakes, rivers, and estuaries. Moreover, the high salt tolerance of Asian clams may allow them to outcompete native clams and mussels in ecosystems contaminated with high concentrations of road salts.


The research, titled “Toxicity of various road‐deicing salts to Asian clams (Corbicula fluminea),” can be found at doi: 10.1002/etc.4126

[David Diehl, site manager and coordinator at the Rensselaer Darrin Fresh Water Institute (DFWI), recently organized a visit by students from Manhattan’s Harlem Academy to learn about the Jefferson Project at Lake George. In this guest post, he chronicles the three-day visit and the topics students learned about. The Jefferson Project is a collaboration between Rensselaer, IBM Research, and The FUND for Lake George, founded to develop a new model for technologically enabled environmental monitoring and prediction to understand and protect the Lake George ecosystem and freshwater ecosystems around the world. This is the third year Harlem Academy has visited DFWI.]

The Jefferson Project at Lake George recently hosted a group of eighth-grade students from Manhattan’s Harlem Academy for three days in which they learned about lake stressors like salt and nutrient loading, and how advanced technology and big data are being used to protect these threats to water quality

The Jefferson Project— a partnership between IBM Research, Rensselaer Polytechnic Institute, and The FUND for Lake George—is building the world’s most advanced environmental monitoring system to better understand freshwater ecosystems and provide science-based insights to decision makers for sustained protection of Lake George, and lakes worldwide. The project’s research team includes more than 100 Rensselaer faculty, staff, and students—in areas including biological sciences, earth and environmental sciences, engineering, and the arts—combined with 20 IBM scientists and engineers and The FUND’s staff and scientific advisers. Since its inception in 2013, the project has built a Smart Sensor Network that gathers more than nine terabytes of physical and chemical data annually; computer models that depict the flow of water and threats throughout the watershed; surveys of plants and animals; and ongoing experiments to determine the impacts of human activities on the lake.

Sixteen students began their visit in May at the Aquatic Research Laboratory of Rick Relyea, the project director. At his lab in Rensselaer’s Tech Park, Relyea’s researchers introduced the Harlem students to the organisms that live in lakes, including zooplankton, macro-invertebrate animals, algae, and aquatic plants. The students explored how these organisms interact in a delicate balance within an aquatic food web, and how the stressors of salt and excess nutrients affect the food web and overall water quality. The students also visited the outdoor mesocosms—large tanks of water that simulate aquatic ecosystems—and they learned how to measure abiotic factors that support aquatic food webs such as dissolved oxygen and sunlight.

On day two, the students visited Rensselaer’s Darrin Fresh Water Institute (DFWI) in Bolton Landing, where they learned about the long history of DFWI researchers studying Lake George and other nearby watersheds. A tour of the labs at the DFWI provided an understanding of how different areas of lake and stream research require specific tools and expertise.

Next, students visited the Helen-Jo and John E. Kelly III ’78 Data Visualization Laboratory at DFWI for an overview of the Jefferson Project, including the computer models used to understand the complex interactions biological and abiotic factors—such as road salt—have on water quality. Students learned how nine terabytes of physical and chemical data collected by the project annually can be coupled with other technologies to control and limit the spread of road salt in sensitive areas or areas that do not need treatment at all.

The students rounded out the day with a tour aboard the Lake George Association’s Floating Classroom. While on the water, students collected zooplankton for microscopic study, conducted water clarity measurements using Secchi discs, and received historical anecdotes involving Lake George in the contexts of the French and Indian War and the American Revolution. On their final day at Rensselaer’s main campus, Harlem students presented their findings and what they learned during their experiences to their classmates.

My own work directing this part of their experience always makes me feel grateful to work with these students, as they always make compelling presentations, and answer thoughtful questions often in humorous and always meaningful ways.

The apparent success of this program would not be possible without the help of Rensselaer’s Cynthia Smith, Rick Relyea, and Kayla Coldsnow, and the Lake George Association’s Walt Lender and Kristen Wilde. Many others are to thank for their help as well.

Credit: Vanessa Wuerthner

[The Jefferson Project at Lake George is conducting ongoing research into how human activities may be affecting the lake. This guest blog by Devin Jones, a former postdoctoral research associate in the lab of Jefferson Project Director Rick Relyea, summarizes recent research published in the journal Environmental Science & Technology. The Jefferson Project is a collaboration between Rensselaer, IBM Research, and The FUND for Lake George, founded to develop a new model for technologically enabled environmental monitoring and prediction to understand and protect the Lake George ecosystem and freshwater ecosystems around the world.]

What did you want to know?

We wanted to explore the origins of “inducible responses” to contaminants in aquatic organisms. When aquatic organisms are exposed to natural chemicals (e.g., those emitted by predators that scare the prey) or human-made chemicals (e.g., pesticides and road salts), these organisms produce short-term physiological, behavioral, or morphological changes that enable them to tolerate their stressful environments. For example, when tadpoles are exposed to a sublethal insecticide concentration, they produce increased levels of an enzyme the insecticide blocks, increasing the tadpoles’ tolerance. How is it that an animal that has never encountered human-made chemicals has developed this adaptation? We hypothesized that responses to natural stressors that have developed over thousands of years might be co-opted to produce adaptive responses to novel contaminants found in water today.

How did you go about it?

We used a two-phase laboratory experiment to investigate the ability of wood frog tadpoles to increase their pesticide tolerance after an early exposure to both natural and human-made stressors. In Phase 1, we exposed tadpoles to one of seven sublethal stressors, including a no-pesticide control, two low concentrations (0.5 and 1.0 milligrams per liter (mg/L )) of the insecticide carbaryl (the active ingredient of the insecticide Sevin®), and two concentrations (200 and 1,000 mg/L) of road salt (NaCl), and natural chemicals emitted by a predatory dragonfly. In Phase 2, tadpoles exposed to each of these seven environments were placed in either a no-insecticide environment, or a high-insecticide environment to investigate changes in pesticide tolerance.

What did you find out?

We found that early exposure to predator chemicals or a low concentration of the insecticide both induced increased tolerance to a subsequent high concentration of the insecticide. This surprising result suggests that the ability of amphibians to respond to low concentrations of pesticides by increasing their tolerance may have arisen through pathways used for responses to predators. Future research investigating the biochemical and physiological processes involved with amphibian responses to predators and pesticides will better provide a mechanism for this novel result.

As humans continue to modify ecosystems, it is likely that organisms will be simultaneously exposed to numerous stressors. The ability of organisms to induce adaptive responses to human-made chemicals might buffer at-risk populations from the lethal effects of contaminants. Given that adaptive responses to natural stressors is common in nature, it is likely that many species possess the ability to ameliorate the negative effects of commonly applied pesticides by quickly increasing their tolerance.

The research titled Inducible tolerance to agrochemicals was paved by evolutionary responses to predators” can be found at DOI: 10.1021/acs.est.7b03816


[The Jefferson Project at Lake George is conducting ongoing research into how human activities may be affecting the lake. This guest blog by Matt Schuler, a postdoctoral research associate in the lab of Jefferson Project Director Rick Relyea, summarizes recent research published in the journal Oikos. The Jefferson Project is a collaboration between Rensselaer, IBM Research, and The FUND for Lake George, founded to develop a new model for technologically enabled environmental monitoring and prediction to understand and protect the Lake George ecosystem and freshwater ecosystems around the world.]

What did you want to know?

Increased application of deicing salts such as sodium chloride (NaCl) has led to higher concentrations of salts in freshwater ecosystems. With the potential for freshwater contamination, and the rising costs of NaCl, agencies are seeking more effective road salts and road salt additives. Magnesium chloride (MgCl2), for example, is more effective than NaCl at colder temperatures and additives such as beet juice make salts stick to roads better. By using these products, agencies need to apply less road salt. However, there is remarkably little information about how these alternative salts and additives might affect freshwater communities. We set out to test how road salt alternatives and additives might affect a common urban mosquito, Culex restuans. Mosquitoes typically have high tolerances to salt, and additives might act as fertilizers that promote more algae, which is consumed by mosquito larvae. Understanding how road salts and salt additives affect mosquito populations could help minimize potential threats to human health.

How did you go about it?

We used outdoor, freshwater mesocosms to experimentally manipulate the concentration of magnesium chloride and two organic additives (beet juice and distillation byproduct). We experimentally tested how three concentrations (low, medium, and high) of magnesium chloride affected mosquito survival, with and without organic additives. We placed 50 freshly hatched mosquito larvae in each mesocosm, and measured daily survival and the time to emergence in each treatment. We also measured important abiotic aspects of the freshwater ecosystem that might affect competitors or predators of mosquitoes, such as the amount of oxygen in the water.

What did you learn?

We found that magnesium chloride reduced mosquito survival at medium and high concentrations, indicating that while mosquitoes are tolerant to sodium chloride, they are not tolerant to all salts. The additives act as fertilizers for the algae, providing more food for the mosquito larvae and thereby reducing the time that it took for mosquito larvae to emerge. Faster emergence of mosquitoes could significantly increase the abundance of mosquitoes, particularly if large quantities of additives are used to make salts stick to roads, but less salt is applied to the roads. Such a regimen would increase the concentration of additives in freshwater systems, but the salt concentration would not increase at the same rate. This would benefit mosquitoes, but might negatively affect their predators and competitors.

Early in the experiment, microbial breakdown of carbohydrates in the additives reduced the oxygen concentration in the water to extremely low levels. Aquatic predators that eat mosquitoes need oxygen in the water to survive. The oxygen concentrations would have been deadly to most fish and insect species that feed on mosquitoes, but would not harm the mosquito species used in this experiment, since they use a syphon tube to breath air. Using additives near freshwater ecosystems might reduce predators, and further increase the abundance of mosquitoes. More comprehensive studies testing the effects of multiple road salts and additives on numerous species of mosquito should be conducted before agencies promote the increased application of alternative road salts and road salt additives.

The research, titled “Road salt and organic additives affect mosquito growth and survival: an emerging problem in wetlands,” can be found at doi:10.1111/oik.04837



Anyone who’s struggled to get their point across in a foreign language they haven’t mastered knows the benefit of enlisting the full contents of the human communication toolbox: waving your arms, imploring with your eyes, describing the word you lack with an assembly of the words you have, even throwing in a little English when nothing else comes to mind is all part of the repertoire. It’s clunky, but our flexible human minds bridge the gap between the ideal words and meaning.

Could a computer do the same? The answer — as found in the Cognitive and Immersive Systems Laboratory (CISL) — is yes. CISL is developing a system that allows Mandarin Chinese language students to practice their new tongue with a cognitive agent. The computational goal is for the agent to understand human communication — including elements like spatial position, gesture, and facial expression — in a specific context, despite deficiencies in vocabulary, grammar, or pronunciation typical of language learners. Ultimately, the system will adapt to students’ ability level, coach them when necessary, and provide feedback on performance and exercises to improve.

Recently, CISL held a test of its “Mandarin Project” system with the help of students enrolled in Chinese 1, a class taught by Yalun Zhou, an expert in second language acquisition and an assistant professor of Communication and Media. The specific context of the conversation was taken from unit 4 of their class — at the restaurant. In Studio 2 of the Curtis R. Priem Experimental Media and Performing Arts Center (EMPAC), students entered a cognitive and immersive environment mimicking a restaurant, greeted a “waiter” (digitally represented as a panda), and tried to order a drink and food.

CISL Director Hui Su, himself a native Mandarin speaker, said the test was intended to collect data on the system’s performance, with a focus on areas where novice Chinese speakers encounter difficulty. By introducing the system to non-native speakers, CISL hopes to uncover patterns in errors, and equip the computer to understand and compensate. In its first trial, Su, a Rensselaer professor of practice in computer science, was impressed with the results:

The dialogue to order food is very simple, but from the perspective of a non-native speakers, it’s already complicated. From greeting, to ordering a drink, to ordering food, and finishing payment is 10 to 20 rounds of conversation with a meaningful task. Nevertheless, all the student pairs were able to greet the waiter and order a drink, a few teams made it through the entire ordering process, and one team completed the payment step, which they haven’t even covered in class. Although we saw a lot of room for improvement, I’m very encouraged by what we’ve achieved thus far.

Zhou, who is aiding CISL in the project, said that for the students, the Mandarin Project represents a unique experience in academia.

The Chinese language learning experience that Studio 2 provides to RPI students is a one-of-a-kind educational experience that their counterparts in other universities cannot match… The ultimate goal of learning a foreign language is to use the target language to communicate with people of that language in a grammatically correct and culturally appropriate way. Research has shown that task-based, communicative language practices lead to the best results for that purpose. The immersive restaurant at Studio 2 provides a near real-life environment for RPI students to practice and test their abilities to complete a communicative task, i.e., ordering Chinese food with the AI waiter whose speech is generated from native speaker accents. This is an invaluable opportunity for the students who are unable to practice speaking with native speakers outside of the classroom.

The Mandarin Project – a reboot of an initiative launched in 2012 to combine narrative, game design, and augmented and virtual reality to teach Chinese — is the latest manifestation of CISL, which is dedicated to pioneering immersive and cognitive systems as an aid to collaborative problem-solving.

A collaboration launched in 2015 between Rensselaer and IBM Research, CISL was charged with developing a cognitive and immersive environment that could be used as a classroom, meeting room, design studio, or diagnosis room. Within a year, CISL had reached its first milestone, completing an initial “cognitive and immersive architecture” that allows a computer to integrate sensory information from multiple sources (like speech, spatial positioning, gesture), translate it into an understanding of events in the room, and offer an appropriate response.

This prototype powered a “meeting room” application run in a cognitive and immersive space in EMPAC Studio 2 that facilitates a business discussion of mergers and acquisitions. Although rudimentary in comparison with human understanding, the initial architecture established a framework that could be continually improved and adapted to other scenarios such as a classroom, design studio, or diagnosis room. The Mandarin Project is one such scenario.

Su divides the capabilities of the Mandarin Project cognitive and immersive system into three “areas.” Area 1 is about “capturing inputs,” such as capturing and combining verbal inputs with gestures, and capturing who is speaking about what, when, to whom, and in what context. Area 2 is about reasoning, planning and understanding — called “Mind-of-the-Room.” And Area 3 focuses on the computer’s meaningful response, which Su calls “immersive narrative generation.”

The Mandarin Project system builds on the initial architecture CISL developed with improvements that can be helpfully divided into the three areas.

For language learners, the most important improvement in Area 1 is a “spatial context system,” which combines gesture and verbal input from multiple channels to generate meaning. As an example, students may point at a menu item and ask “waiter, what is this?” without specifying the meaning of “this” (a word they may not know), and the system will know what they mean.

Area 2 includes a lot of what Su calls “help functions.” A “request cue” feature allows users to ask in English or Chinese “what am I supposed to say next?” or “what did you say?” and the system will respond appropriately. Language switching allows users to request to “switch back to English” when needed. Users may request a transcription of input the system recognized, for their own review.

The Mandarin Project also has several help functions under development that it did not test, including a “pitch tone contour analysis.” Chinese syllables can be pronounced with four “tones,” each of which alters the meaning of the syllable and ultimately the word, and correct tone is often a daunting skill for students to master. As an aid to improve pronunciation, a “visual tone cue” will display the correct tone above words that are part of the dialogue. The pitch tone contour analysis will be used to record the tones a user used, and how that compares to the correct tone choice, showing users where they erred and how to improve.

Su says developers are “just getting started” on Area 2, with several additional features planned. Among their ideas, “adaptive response” will track use of help functions, and adjust difficulty of the dialogue accordingly. A “tone help” function will offer correct verbal pronunciations of words that are mis-pronounced. An “alternate approach” will understand users who try to describe a word they don’t know with the words they do know. And a “personalized learning plan” function will generate exercises for each user based on their performance.

Some Area 2 improvements are dedicated to adapting a Chinese language speech recognition engine that was created for native users. The system was “trained” on native speakers, and tolerances for tone and pronunciation are set accordingly. But those settings can be altered for non-native speakers based on the experiences in Studio 2.

In Area 3, immersive narrative generation, the system is able to offer information about specific dishes on the menu, like the history of Peking duck. The system is able to fetch data from DBPedia, although at the moment information is pre-fed into the computer. Eventually, the system will fetch information in real time.

Although the system is still very much a prototype, Zhou said she is certain her students are the beneficiaries.

The students enjoyed the immersive interaction and expressed that the immersive learning allowed them to gain an understanding of how their speaking ability is developing/progressing. From the point view of the instructor, these “smart” technologies of the three areas are appealing because they “force” the students to interact with the AI waiter like a real customer in an authentic Chinese restaurant. The multimodal multitasking (e.g., listening and speaking to the waiter and reading the menu simultaneously, reading the transcription to diagnose errors and adjusting their speaking, and the instant help function) stimulates their desire to complete the food ordering task in real time. Although we are still constrained by the available technology for full function of pedagogical design, as a foreign language educator, I am thrilled to see how novice learners can practice in a simulated restaurant, be motivated to complete the communicative task, and increase their level of confidence regarding speaking.

[The Jefferson Project at Lake George is conducting ongoing research into how human activities may be affecting the lake. This guest blog by Bill Hintz, a post-doctoral research associate in the lab of Jefferson Project Director Rick Relyea, summarizes recent research published in the journal Oecologia. The Jefferson Project is a collaboration between Rensselaer, IBM Research, and The FUND for Lake George, founded to develop a new model for technologically enabled environmental monitoring and prediction to understand and protect the Lake George ecosystem and freshwater ecosystems around the world.]

What did you want to know?

Fear of predators stresses out prey. This so-called “landscape of fear” can affect prey by reducing their growth, altering their reproductive patterns, and changing their behavior. Research has shown that pollution can have the same effect. In fact, the natural stress caused by predators can interact with pollution. For example, combining a non-lethal concentration of pesticides with predatory stress can be lethal to many freshwater organisms. Road salt is a common pollutant in cold regions that can dramatically increase the salinity of freshwater ecosystems. We wanted to know if pollution from road salt interacted with predatory stress in freshwater systems to alter the reproduction, behavior, and growth of fish and zooplankton.

How did you go about it?

We conducted two experiments, one on rainbow trout and another on the zooplankton species Daphnia pulex. Rainbow trout are an important sport fish around the world and Daphnia help keep water in our lakes and wetlands clear by consuming algae. In the trout experiment, we exposed them to three concentrations of road salt crossed with the presence or absence of an alarm cue—essentially a “smell” in the water that prey produce when they are eaten by predators. Similarly, we exposed Daphnia to four concentrations of road salt and the alarm cue produced when predators consume Daphnia.

What did you find out?

In the Daphnia experiment, we found that alarm cues and road salt had a combined effect on Daphnia reproduction. In the absence of road salt, the presence of alarm cues induced Daphnia to produce eggs, which is a normal response to the stress of predators. Daphnia produced more eggs as salt concentrations increased, an indication that Daphnia respond to the presence of salt as a stress, similar to how they respond to the stress of predators. However, when the stress of predators and salt are combined, egg production did not increase as expected, indicating that elevated salt levels inhibited Daphnia’s ability to respond to predators.

However, while Daphnia abundance—i.e., population size—was independently affected by both road salt and the presence of alarm cues, we did not observe a combined effect. High concentrations of road salt reduced Daphnia abundance by 85 percent, while alarm cues reduced abundance by 11 percent.

We found that the rainbow trout were affected by both the alarm cue and salt, but we did not see a combined effect. Regardless of salt concentrations, the alarm cue triggered behavioral changes, causing the trout to form groups, decrease their movement behavior, and reduce their aggression toward each other. Regardless of alarm cues being present or absent, road salt reduced trout growth at high concentrations compared with moderate concentrations.

Our results indicate that predatory stress and road salts can both affect freshwater organisms in ways that could alter the food webs of freshwater ecosystems. The salt concentrations that caused harm to the two species were nearly 100 times higher than salt concentrations in Lake George, but similar to the pulses of salt observed in the streams that supply water to the lake.

The research, titled A salty landscape of fear: responses of fish and zooplankton to freshwater salinization and predatory stress,” can be found at doi: 10.1007/s00442-017-3925-1.


How is it that a shark can sense electric fields generated by its prey? To find out, glycoproteins expert Robert Linhardt turned to an instrument called a MALDI TOF TOF mass spectrometer in the Rensselaer Center for Biotechnology and Interdisciplinary Studies (CBIS), and in this post we’re going to talk about MALDI TOF TOF (we’ll get to the name), how it works, and how that makes it a valuable instrument in translational medicine.

But first, back to sharks. Sharks use an organelle – called the ampullae of Lorenzini – lining their heads to sense electrical fields generated by fish and other animals around them in the water. To understand how the ampullae of Lorenzini translates electrical fields in the water to electrical signals in their brain, Linhardt first needs a full accounting of the proteins in the organelle.

One way to compile that list would be to look at the shark’s genome, in which are encoded directions for all the proteins the shark can produce. But in this case, the shark’s genome hasn’t been sequenced. So the next best thing is to sample and identify each protein present in the organelle. The MALDI provides a rapid, semi-quantitative structure and sequence of each protein, allowing researchers to compare what they’ve found to well-documented proteins in a related, better known, organisms, and compile their list. Here’s how Linhardt puts it:

When you have a biological organ and you want to understand what proteins are present, you do MALDI TOF TOF. Without a lot of set-up, MALDI gives you a really quick interpretation of what proteins are present. If you have a lot of samples, you put them on the target and you get a lot of results. It’s not ultra-sensitive, but it gives enough information to explain the structure of the proteins that are there and some of their interesting properties.

And how does it do that? Like other forms of mass spectrometry, MALDI TOF TOF takes advantage of the simple truth that atoms of each element, as well as ions and isotopes of those elements, have a unique mass. Therefore molecules – made up of atoms, ions, and isotopes – also have a unique mass.

Mass spectrometers in general determine that mass by propelling a molecule through a vacuum using an electric or magnetic field, and obtaining data that can be translated into the identity of the chemical species. But to enter the vacuum the molecule must be evaporated into the gas state and ionized (possess a positive or negative charge). A variety of instruments have been developed that best analyze particular classes of chemicals by using different techniques to vaporize, ionize, and measure the mass of the molecule.

And so while MALDI TOF TOF is a mass spectrometer, it’s full name — Matrix Assisted Laser Desorption/Ionization Time-of-Flight Time-of-Flight — indicates the ionization and vaporization as well as how masses of ions are measured.

Dmitri Zagorevski, director of the Proteomics Core, where the MALDI lives, explained that in MALDI, samples to be tested are mixed in a large quantity of matrix, a liquid typically consisting of crystallized molecules, such as sinapinic acid. A droplet of the sample in matrix is dropped onto a metal plate and loaded into the instrument. When a laser is pulsed at the droplet, most of its energy is absorbed by the matrix, which vaporizes together with the target sample. Molecules in the plume of resulting gas are ionized and then accelerated into the vacuum. The instrument measures the time of flight across a fixed distance in two directions, hence Time-of-Flight Time-of-Flight.

As a vaporization and ionization technique, MALDI is well suited for analysis of fragile organic molecules – particularly non-volatile biological molecules like proteins, peptides, and oligosaccharides – that would disintegrate under conventional ionization methods.

The MALDI is one of four mass spectrometers in the Proteomics Core, which is itself one of ten research cores – each housing specialized equipment and facilities — within CBIS. Each of the mass spectrometers is suited to a specific role and each supports research that is elucidating the complexities of biological processes, like our circadian rhythms, tracking down the causes of diseases like Alzheimer’s, and producing innovations in areas such as regenerative medicine.

For some researchers, like Wilfredo Colón, professor and head of the Department of Chemistry and Chemical Biology, MALDI is indispensable. Colón uses the MALDI to identify hyperstable proteins in various organisms and biological systems. Hyperstable (i.e., difficult to degrade) proteins play important biological and pathological roles, and the Colón lab has developed a unique technique that isolates these proteins, allowing them to be identified via MALDI. Without the MALDI, his work wouldn’t be possible. Here’s how he put it:

As useful as our method is for separating hyperstable proteins from most of the proteins in a sample, without MALDI we would not be able to quickly identify the proteins, and this is where the exciting discoveries lie. Our research without MALDI is like getting a Christmas gift and not knowing what’s inside – the latter is nice, but unwrapping the gift is the exciting part. MALDI is already yielding exciting “knowledge gifts” in our research and more are on the way.

A MALDI instrument was part of the original suite of equipment installed in CBIS when it opened in 2004. It’s the only one of its kind at Rensselaer, and one of just a few in the Capital Region. Recently, the original MALDI was replaced with a newer, faster, and better version. The CBIS website lists the other instruments within the core as: a Thermo LTQ-Orbitrap mass spectrometer and a Thermo TSQ Quadrupole mass spectrometer, both coupled with micro-flow Agilent HPLC systems, and a Shimadzu gas chromatograph mass spectrometer.

In addition to the Proteomics Core Facility, CBIS has additional research equipment and facilities in analytical biochemistry and nanobiotechnology, bioimaging, bioresearch, cell and molecular biology, flow cytometry, microbiology and fermentation, microscopy, nuclear magnetic resonance, and stem cell research. The cores are available to Rensselaer faculty, staff and students, and also to external academic and industrial collaborators and researchers. As one of the most advanced research facilities in the nation, CBIS promotes innovation and discovery at the interface of the life sciences, physical sciences, and engineering.