Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Final arm64 related functionality #1087

Merged
merged 12 commits into from May 16, 2019
Merged

Conversation

@dianpopa
Copy link
Contributor

@dianpopa dianpopa commented May 9, 2019

With this functionality, I can ssh into an aarch64 guest.
Replication steps to come.

Still some bugs need to be solved:

  • time is wrong
  • seccomp filters are not applied

Issue #, if available: Fixes #757.

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copy link
Contributor

@petreeftime petreeftime left a comment

The commit message on commit af4d5a0 looks incomplete.

guest_mac: None,
rx_rate_limiter: None,
tx_rate_limiter: None,
allow_mmds_requests: false,
tap: None,
};
assert!(vmm.insert_net_device(network_interface).is_ok());
vmm.insert_net_device(network_interface).expect("lala");

This comment has been minimized.

@petreeftime

petreeftime May 9, 2019
Contributor

The expect message is not very helpful.

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

Sorry, some leftover debug messages.


self.configure_system()?;
#[cfg(target_arch = "aarch64")]

This comment has been minimized.

@petreeftime

petreeftime May 9, 2019
Contributor

Maybe it's explained in the missing part of the commit message, but it's not clear why these two setup steps are so different. Is it an error on x86_64 to create the vcpus earlier? Why can't the first step be load_kernel on both?

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

I updated the commit message and also put a comment before branching the implementation. Take a look.

This comment has been minimized.

@petreeftime

petreeftime May 14, 2019
Contributor

Looks much cleaner now.

Could it be possible to further reduce duplication by doing the following:

            let entry_addr = self.load_kernel()?;

            #[cfg(target_arch = "aarch64")]
            vcpus = self.create_vcpus(entry_addr, request_ts)?;

            self.setup_interrupt_controller()?;
            self.attach_virtio_devices()?;
            self.attach_legacy_devices()?;

            #[cfg(target_arch = "x86_64")]
            vcpus = self.create_vcpus(entry_addr, request_ts)?;

This comment has been minimized.

@acatangiu

acatangiu May 15, 2019
Contributor

nit: Would be nice to also add a comment for both like:

// On aarch64 we need to create vcpus before creating the interrupt controller.
#[cfg(target_arch = "aarch64")]
vcpus = self.create_vcpus(entry_addr, request_ts)?;
...
// On x86_64 we need to create vcpus after setting up everything else.
#[cfg(target_arch = "x86_64")]
vcpus = self.create_vcpus(entry_addr, request_ts)?;

(Or smth similar)

self.attach_virtio_devices()?;
self.attach_legacy_devices()?;

self.register_events()?;

This comment has been minimized.

@petreeftime

petreeftime May 9, 2019
Contributor

Why are register_events and configure_system swapped?

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

No reason and I also just realized that there is no reason in having them inside the aarch64 cfg. I removed them from there. Take a look.

Copy link
Contributor

@acatangiu acatangiu left a comment

Really nice structure and separation of ARM and X86_64, cool! 👍

@@ -272,7 +283,7 @@ mod tests {
let intr_evt = EventFd::new().unwrap();
let serial_out = SharedBuffer::new();

let mut serial = Serial::new_out(intr_evt, Box::new(serial_out.clone()));
let mut serial = Serial::new_out(intr_evt, Box::new(serial_out.clone()), None);

This comment has been minimized.

@acatangiu

acatangiu May 10, 2019
Contributor

Is this test not run on aarch64? Wouldn't it need len=4 on arm?

Same question for all following tests.

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

It is run on aarch64 too, but given that this is an unit test it will hit the same code like on x86.
What makes a difference here is the enable_earlycon (from a later commit) function that indeed is specific to aarch64 (not even specific it is just that on aarch64 we need to do it through MMIO addreses os the data.len will be 4).
TO be honest, what I see worth unit testing here is checking that the METRICS.uart.missed_read_count.inc();
stays 0 after a read that has the same length as the length to what the device was instructed to accept inside the constructor.
Another thing that we could test here is that when we initiliaze the length to some different length different than 1, the device knows to read data only of that length.

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

Exercised all of the above with test_serial_data_len. Take a look.

addr: u64,
irq: u32,
len: u64,
type_: u32,

This comment has been minimized.

@acatangiu

acatangiu May 10, 2019
Contributor

nit: Maybe dev_type instead of type_?

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

I thought about this but dev_type inside MMIODeviceInfo?

This comment has been minimized.

@acatangiu

acatangiu May 15, 2019
Contributor

I thought about this but dev_type inside MMIODeviceInfo?

Sounds good to me 😆, but like I said, it's a nit.

#[derive(Clone, Debug)]
pub struct MMIODeviceInfo {
addr: u64,
irq: u32,

This comment has been minimized.

@acatangiu

acatangiu May 10, 2019
Contributor

nit: reorder suggestion: put len right after addr since it's a familiar pattern.

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

I tried to maintain alphabetical order :(

This comment has been minimized.

@acatangiu

acatangiu May 15, 2019
Contributor

Still a nit, but addr and len are logically tied, they describe the device address space slice, and that trumps alphabetical order for me.


if let Some(device_id) = id {
self.id_to_addr_map.insert(device_id.clone(), ret);
self.id_to_dev_info.insert(
device_id.clone(),

This comment has been minimized.

@acatangiu

acatangiu May 10, 2019
Contributor

Doesn't need cloning. You're already cloning when calling this function. Either:

  • make id param as id: Option<&String> and clone it here (callers of the func would not clone id anymore),
    or
  • leave id param as it is, and consume it here instead of cloning.

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

Done, I actually made id mandatory (not an Option anymore). Leaving it as Option made the code very prone to programming errors since on aarch64, non mentioning an id would not make it part of the FDT. Take a look.

fn addr(&self) -> u64;
fn irq(&self) -> u32;
fn length(&self) -> u64;
fn type_(&self) -> u32;

This comment has been minimized.

@acatangiu

acatangiu May 10, 2019
Contributor

type_ would look better as an enum clearly stating virtio or serial instead of an u32 on which we map certain numbers to certain types.

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

Done, take a look.

.unwrap();
output.write_all(&dtb).unwrap();
}
{

This comment has been minimized.

@acatangiu

acatangiu May 10, 2019
Contributor

nit: looks like some extra indentation slipped in here

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

Done.

#[cfg(target_arch = "aarch64")]
/// Gets the information of the devices registered up to some point in time.
pub fn get_device_info(&self) -> HashMap<String, MMIODeviceInfo> {
self.id_to_dev_info.clone()

This comment has been minimized.

@acatangiu

acatangiu May 10, 2019
Contributor

Haven't looked at the implications, but would like to see this returning a reference instead of a clone.

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

I can indeed return a reference here. Take a look.

@@ -1162,17 +1162,9 @@ impl Vmm {
.ok_or(StartMicrovmError::VcpusNotConfigured)?;

This comment has been minimized.

@acatangiu

acatangiu May 10, 2019
Contributor

Commit af4d5a0b292dfc6ef6b381d75f76c4f5ff2c3701 has an incomplete description.

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

Solved.

guest_mac: None,
rx_rate_limiter: None,
tx_rate_limiter: None,
allow_mmds_requests: false,
tap: None,
};
assert!(vmm.insert_net_device(network_interface).is_ok());
vmm.insert_net_device(network_interface).expect("lala");
//assert!(vmm.insert_net_device(network_interface).is_ok());

This comment has been minimized.

@acatangiu

acatangiu May 10, 2019
Contributor

Leftover commented line.

This comment has been minimized.

@dianpopa

dianpopa May 14, 2019
Author Contributor

Solved.

@dianpopa dianpopa force-pushed the dianpopa:i757k branch from aababb0 to 3ad9598 May 14, 2019
Copy link
Contributor

@petreeftime petreeftime left a comment

LGTM.

fn length(&self) -> u64 {
self.len
}
fn type_(&self) -> FDTDeviceType {

This comment has been minimized.

@acatangiu

acatangiu May 15, 2019
Contributor

What do you think about making type_ in MMIODeviceInfo also an enum, ideally the same enum as FDTDeviceType.

I think you can call this something generic (not fdt related) and define it for both x86_64 and aarch64 in order to use it everywhere.

This comment has been minimized.

@dianpopa

dianpopa May 16, 2019
Author Contributor

Good suggestion. Take a look.

dianpopa added 7 commits May 1, 2019
The register_device function inside the mmio device manager actually
adds a virtio device. You can also have mmio devices added to the bus
without them actually using virtio protocol.

Signed-off-by: Diana Popa <dpopa@amazon.com>
By default our console implementation is hardcoded to read 1 byte
since we expect the data to come from a I/O port. However, when
implementing early console support through MMIO addresses, the length of bytes
received at that address is 4.

Signed-off-by: Diana Popa <dpopa@amazon.com>
We need it for loading the kernel for each architecture.

Signed-off-by: Diana Popa <dpopa@amazon.com>
CMDLINE_MAX_ADDR was removed from common interface
since loading the commandline to a specific address
is x86_64 specific.

Signed-off-by: Diana Popa <dpopa@amazon.com>
Signed-off-by: Diana Popa <dpopa@amazon.com>
Up until now we could only obtain the address where some mmio device
had been registered. For creating the Flattened Device Tree we also need
the length and the irq associated with it.

Signed-off-by: Diana Popa <dpopa@amazon.com>
aarch64 has no I/O port support so we cannot enable serial
the same way we do for x86_64. We are enabling it by adding an
earlycon device at some MMIO address.

Signed-off-by: Diana Popa <dpopa@amazon.com>
@dianpopa dianpopa dismissed stale reviews from acatangiu and petreeftime via 66ebd60 May 15, 2019
@dianpopa dianpopa force-pushed the dianpopa:i757k branch from 3ad9598 to 66ebd60 May 15, 2019
dianpopa added 5 commits Apr 30, 2019
Signed-off-by: Diana Popa <dpopa@amazon.com>
On aarch64, KVM_CREATE_VCPUS needs to be called
before setting up the interrupt controller (i.e GIC) so
the sequence of start operations was adjusted to fit
kernel requirements.

Signed-off-by: Diana Popa <dpopa@amazon.com>
By default, on aarch64 the rootfs is mounted in readonly mode
if not specified otherwise.

Signed-off-by: Diana Popa <dpopa@amazon.com>
Signed-off-by: Diana Popa <dpopa@amazon.com>
Signed-off-by: Diana Popa <dpopa@amazon.com>
@dianpopa dianpopa force-pushed the dianpopa:i757k branch from 66ebd60 to d609bfe May 16, 2019
Copy link
Contributor

@acatangiu acatangiu left a comment

💯

@dianpopa dianpopa merged commit 10fdf2d into firecracker-microvm:master May 16, 2019
1 check passed
1 check passed
default All checks have passed. Check the results in the target URL.
Details
@dianpopa dianpopa deleted the dianpopa:i757k branch May 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants
You can’t perform that action at this time.