Friday, January 4, 2019

Artificial intelligence turns brain activity into speech

Artificial intelligence turns brain activity into speech. Kelly Servick. Jan. 2, 2019.

For many people who are paralyzed and unable to speak, signals of what they'd like to say hide in their brains. No one has been able to decipher those signals directly. But three research teams recently made progress in turning data from electrodes surgically placed on the brain into computer-generated speech. Using computational models known as neural networks, they reconstructed words and sentences that were, in some cases, intelligible to human listeners.

None of the efforts, described in papers in recent months on the preprint server bioRxiv, managed to re-create speech that people had merely imagined. Instead, the researchers monitored parts of the brain as people either read aloud, silently mouthed speech, or listened to recordings. But showing the reconstructed speech is understandable is "definitely exciting," says Stephanie Martin, a neural engineer at the University of Geneva in Switzerland who was not involved in the new projects.

People who have lost the ability to speak after a stroke or disease can use their eyes or make other small movements to control a cursor or select on-screen letters. (Cosmologist Stephen Hawking tensed his cheek to trigger a switch mounted on his glasses.) But if a brain-computer interface could re-create their speech directly, they might regain much more: control over tone and inflection, for example, or the ability to interject in a fast-moving conversation.

The hurdles are high. "We are trying to work out the pattern of … neurons that turn on and off at different time points, and infer the speech sound," says Nima Mesgarani, a computer scientist at Columbia University. "The mapping from one to the other is not very straightforward." How these signals translate to speech sounds varies from person to person, so computer models must be "trained" on each individual. And the models do best with extremely precise data, which requires opening the skull.

Researchers can do such invasive recording only in rare cases. One is during the removal of a brain tumor, when electrical readouts from the exposed brain help surgeons locate and avoid key speech and motor areas. Another is when a person with epilepsy is implanted with electrodes for several days to pinpoint the origin of seizures before surgical treatment. "We have, at maximum, 20 minutes, maybe 30," for data collection, Martin says. "We're really, really limited."

The groups behind the new papers made the most of precious data by feeding the information into neural networks, which process complex patterns by passing information through layers of computational "nodes." The networks learn by adjusting connections between nodes. In the experiments, networks were exposed to recordings of speech that a person produced or heard and data on simultaneous brain activity.

Mesgarani's team relied on data from five people with epilepsy. Their network analyzed recordings from the auditory cortex (which is active during both speech and listening) as those patients heard recordings of stories and people naming digits from zero to nine. The computer then reconstructed spoken numbers from neural data alone; when the computer "spoke" the numbers, a group of listeners named them with 75% accuracy.

[recording: A computer reconstruction based on brain activity recorded while a person listened to spoken digits.
H. Akbari et al.,]

Another team, led by neuroscientists Miguel Angrick of the University of Bremen in Germany and Christian Herff at Maastricht University in the Netherlands, relied on data from six people undergoing brain tumor surgery. A microphone captured their voices as they read single-syllable words aloud. Meanwhile, electrodes recorded from the brain's speech planning areas and motor areas, which send commands to the vocal tract to articulate words. The network mapped electrode readouts to the audio recordings, and then reconstructed words from previously unseen brain data. According to a computerized scoring system, about 40% of the computer-generated words were understandable.

[recording: Original audio from a study participant, followed by a computer recreation of each word, based on activity in speech planning and motor areas of the brain.
M. Angrick et al.,]

Finally, neurosurgeon Edward Chang and his team at the University of California, San Francisco, reconstructed entire sentences from brain activity captured from speech and motor areas while three epilepsy patients read aloud. In an online test, 166 people heard one of the sentences and had to select it from among 10 written choices. Some sentences were correctly identified more than 80% of the time. The researchers also pushed the model further: They used it to re-create sentences from data recorded while people silently mouthed words. That's an important result, Herff says—"one step closer to the speech prosthesis that we all have in mind."

However, "What we're really waiting for is how [these methods] are going to do when the patients can't speak," says Stephanie Ri├Ęs, a neuroscientist at San Diego State University in California who studies language production. The brain signals when a person silently "speaks" or "hears" their voice in their head aren't identical to signals of speech or hearing. Without external sound to match to brain activity, it may be hard for a computer even to sort out where inner speech starts and ends.

Decoding imagined speech will require "a huge jump," says Gerwin Schalk, a neuroengineer at the National Center for Adaptive Neurotechnologies at the New York State Department of Health in Albany. "It's really unclear how to do that at all."

One approach, Herff says, might be to give feedback to the user of the brain-computer interface: If they can hear the computer's speech interpretation in real time, they may be able to adjust their thoughts to get the result they want. With enough training of both users and neural networks, brain and computer might meet in the middle.


[Recordings at the link above at the beginning]

Check also Towards reconstructing intelligible speech from the human auditory cortex. Hassan Akbari, Bahar Khalighinejad, Jose L. Herrero, Ashesh D. Mehta & Nima Mesgaran. Scientific Reportsvolume 9, Article number: 874 (2019),

Photosynthesis may produce toxic by-products that reduce its efficiency; a transgenic plant efficiently recaptures the unproductive by-products of photosynthesis with less energy lost, plants grew ∼40% more

Synthetic glycolate metabolism pathways stimulate crop growth and productivity in the field. Paul F. South, Amanda  P. Cavanagh, Helen W. Liu, Donald R. Ort. Science  Jan 04 2019, Vol. 363, Issue 6422, eaat9077, DOI: 10.1126/science.aat9077

Fixing photosynthetic inefficiencies: In some of our most useful crops (such as rice and wheat), photosynthesis produces toxic by-products that reduce its efficiency. Photorespiration deals with these by-products, converting them into metabolically useful components, but at the cost of energy lost. South et al. constructed a metabolic pathway in transgenic tobacco plants that more efficiently recaptures the unproductive by-products of photosynthesis with less energy lost (see the Perspective by Eisenhut and Weber). In field trials, these transgenic tobacco plants were ∼40% more productive than wild-type tobacco plants.

Structured Abstract
INTRODUCTION: Meeting food demands for the growing global human population requires improving crop productivity, and large gains are possible through enhancing photosynthetic efficiency. Photosynthesis requires the carboxylation of ribulose-1,5-bisphosphate (RuBP) by ribulose-1,5-bisphosphate carboxylase-oxygenase (RuBisCO), but photorespiration occurs in most plants such as soybean, rice, and wheat (known as C3 crops) when RuBisCO oxygenates RuBP instead, requiring costly processing of toxic byproducts such as glycolate. Photorespiration can reduce C3 crop photosynthetic efficiency by 20 to 50%. Although various strategies exist for lowering the costs of photorespiration, chamber- and greenhouse-grown plants with altered photorespiratory pathways within the chloroplast have shown promising results, including increased photosynthetic rates and plant size.

RATIONALE: To determine if alternative photorespiratory pathways could effectively improve C3 field crop productivity, we tested the performance of three alternative photorespiratory pathways in field-grown tobacco. One pathway used five genes from the Escherichia coli glycolate oxidation pathway; a second pathway used glycolate oxidase and malate synthase from plants and catalase from E. coli; and the third pathway used plant malate synthase and a green algal glycolate dehydrogenase. All enzymes in the alternative pathway designs were directed to the chloroplast. RNA interference (RNAi) was also used to down-regulate a native chloroplast glycolate transporter in the photorespiratory pathway, thereby limiting metabolite flux through the native pathway. The three pathways were introduced with and without the transporter RNAi construct into tobacco, which is an ideal model field crop because it is easily transformed, has a short life cycle, produces large quantities of seed, and develops a robust canopy similar to that of other field crops.

RESULTS: Using a synthetic biology approach to vary promoter gene combinations, we generated a total of 17 construct designs of the three pathways with and without the transporter RNAi construct. Initial screens for photoprotection by alternative pathway function under high–photorespiratory stress conditions identified three to five independent transformants of each design for further analysis. Gene and protein expression analyses confirmed expression of the introduced genes and suppression of the native transporter in RNAi plants. In greenhouse screens, pathway 1 increased biomass by nearly 13%. Pathway 2 showed no benefit compared to wild type. Introduction of pathway 3 increased biomass by 18% without RNAi and 24% with RNAi, which were consistent with changes in photorespiratory metabolism and higher photosynthetic rates. Ultimately, field testing across two different growing seasons showed >25% increase in biomass of pathway 3 plants compared to wild type, and with RNAi productivity increased by >40%. In addition, this pathway increased the light-use efficiency of photosynthesis by 17% in the field.

CONCLUSION: Engineering more efficient photorespiratory pathways into tobacco while inhibiting the native pathway markedly increased both photosynthetic efficiency and vegetative biomass. We are optimistic that similar gains may be achieved and translated into increased yield in C3 grain crops because photorespiration is common to all C3 plants and higher photosynthetic rates under elevated CO2, which suppresses photorespiration and increases harvestable yield in C3 crops.

We test the reproducibility/generalisability of priming effects on risk attitudes: we find no systematic effect; fear does not appear to cause countercyclical risk aversion; positive affect makes participants take more risk

On the priming of risk preferences: The role of fear and general affect. Despoina Alempaki, Chris Starmer, Fabio Tufano. Journal of Economic Psychology,

•    We test the reproducibility/generalisability of priming effects on risk attitudes.
•    In a series of experiments, we sample over 1900 subjects from a diverse population.
•    We find no systematic effect of priming on individuals’ risk attitudes.
•    Fear does not appear to cause countercyclical risk aversion.
•    Positive affect makes participants take more risk.

Abstract: Priming is an established tool in psychology for investigating aspects of cognitive processes underlying decision making and is increasingly applied in economics. We report a systematic attempt to test the reproducibility and generalisability of priming effects on risk attitudes in a more diverse population than professionals and students, when priming using either a positive or a negative experience. We further test fear as the causal mechanism underlying countercyclical risk aversion. Across a series of experiments with a total sample of over 1900 participants, we are unable to find any systematic effect of priming on risk preferences. Moreover, our results challenge the role of fear as the mechanism underlying countercyclical risk aversion; we find evidence of an impact of general affect such that the better our participants feel, the more risk they take.

Pattern Analysis of World Conflicts over the past 600 years: War is a statistical phenomenon related to the network structure of the human society

Pattern Analysis of World Conflicts over the past 600 years. Gianluca Martelloni, Francesca Di Patti, Ugo Bardi. arXiv, Dec 2018.

Abstract: We analyze the database prepared by Brecke (Brecke 2011) for violent conflict, covering some 600 years of human history. After normalizing the data for the global human population, we find that the number of casualties tends to follow a power law over the whole data series for the period considered, with no evidence of periodicity. We also observe that the number of conflicts, again normalized for the human population, show a decreasing trend as a function of time. Our result agree with previous analyses on this subject and tend to support the idea that war is a statistical phenomenon related to the network structure of the human society.

Advisors interpersonally penalize those who do not follow their advice; penalize seekers more than non-expert advisors; interpersonally penalize seekers consulting multiple advisors

Seeker beware: The interpersonal costs of ignoring advice. Hayley Blunden et al. Organizational Behavior and Human Decision Processes, Volume 150, January 2019, Pages 83-100.

•    Advisors interpersonally penalize those who do not follow their advice.
•    Expert advisors penalize seekers more than non-expert advisors.
•    Advisors interpersonally penalize seekers consulting multiple advisor.
•    Advisors to “multiple advice-seekers” perceive their advice will be disregarded.
•    Advice seekers view the purpose of seeking advice as gathering information.
•    Advisors (vs. seekers) view their purpose as more to provide direction.
•    Seekers should consider interpersonal goals alongside decision accuracy ones.

Abstract: Prior advice research has focused on why people rely on (or ignore) advice and its impact on judgment accuracy. We expand the consideration of advice-seeking outcomes by investigating the interpersonal consequences of advice seekers’ decisions. Across nine studies, we show that advisors interpersonally penalize seekers who disregard their advice, and that these reactions are especially strong among expert advisors. This penalty also drives advisor reactions to a widely-recommended advice-seeking strategy: soliciting multiple advisors to leverage the wisdom of crowds. Advisors denigrate and distance themselves from seekers who they learn consulted others, an effect mediated by perceptions that their own advice will be disregarded. Underlying these effects is an asymmetry between advisors’ and seekers’ beliefs about the purpose of the advice exchange: whereas advisors believe giving advice is more about narrowing the option set by providing direction, seekers believe soliciting advice is more about widening the option set by gathering information.

How husbands and wives report their earnings when she earns more: Inflating their reports of husbands’ earnings & deflating their reports of wives’ earnings

Manning up and womaning down: How husbands and wives report their earnings when she earns more. Marta Murray-Close and Misty L. Heggeness, US Census Bureau, Working Paper Number SEHSD-WP2018-20,

Abstract: Do gendered social norms influence survey reports of “objective” economic outcomes? This paper compares the earnings reported for husbands and wives in the Current Population Survey with their “true” earnings from administrative income-tax records. Estimates from OLS regressions show that survey respondents react to violations of the norm that husbands earn more than their wives by inflating their reports of husbands’ earnings and deflating their reports of wives’ earnings. On average, the gap between a husband’s survey and administrative earnings is 2.9 percentage points higher if his wife earns more than he does, and the gap between a wife’s survey and administrative earnings in 1.5 percentage points lower if she earns more than her husband does. These findings suggest that gendered social norms can influence survey reports of seemingly objective outcomes and that their impact may be heterogeneous not just between genders but also within gender.

We focus on specific trading days on which investors are primed for honest behavior (Yom Kippur); the patterns found may reflect the increased awareness to honesty

The High Holidays: Psychological Mechanisms of Honesty in Real-Life Financial Decisions. Doron Kliger, Mahmoud Qadan. Journal of Behavioral and Experimental Economics, Journal of Behavioral and Experimental Economics,

•    We focus on specific trading days on which investors are primed for honest behavior
•    Returns during these ten days (High Holidays) are abnormally low; and
•    Implied volatility, as well as realized volatility estimates are abnormally high
•    These systematic patterns may reflect the increased awareness to honesty
•    We suggest a simple trading rule that investors may utilize during these days

ABSTRACT: Research in psychology has established that activation of religious ideas affects individuals’ behavior. We hypothesize that religious and honesty mechanisms activated on the High Holidays, the ten days before Yom Kippur, when people seek repentance, amplify people's anxiety and affect their financial decision-making. We find that returns during the High Holidays are abnormally low; implied volatility, measured by VIX and VXO, as well as realized volatility estimates, are abnormally high; and the abnormal increase in implied volatility overshoots future volatility. Using these results, we devise a simple trading rule that investors may consider to maximize returns during the High-Holidays period.