Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-10-13
09:38:32 lyarwood ah wait, the filename needs to be test_
09:39:37 gibi lyarwood: don't be too hard on yourself (and the keyboard)
09:40:16 lyarwood gibi: ^_^
14:06:43 openstackgerrit Rodrigo Barbieri proposed openstack/nova stable/train: Allow tap interface with multiqueue https://review.opendev.org/757870
14:32:23 mnaser sean-k-mooney: have you played around with sriov + mlx cards?
14:32:33 mnaser i'm having instance creation failures with errors like this: `group 13 is not viable\nPlease ensure all devices within the iommu_group are bound to their vfio bus driver.`
14:32:45 sean-k-mooney mnaser: stephenfin has they are more work then intel
14:33:11 sean-k-mooney mnaser: both prots on the same physical card i think are in the same iommue group
14:33:28 sean-k-mooney you can check that with sysfs
14:33:39 sean-k-mooney mnaser:you need to have both port bound to the same driver
14:35:56 mnaser sean-k-mooney: not sure i follow, but, it seems like they are all in group 13 according to `/sys/kernel/iommu_groups/`
14:36:50 mnaser qemu-system-x86_64: -device vfio-pci,host=0000:41:00.5,id=hostdev0,bus=pci.0,addr=0x4: vfio 0000:41:00.5: group 13 is not viable\nPlease ensure all devices within the iommu_group are bound to their vfio bus driver.
14:37:07 mnaser i wonder if i need to setup vfio stuff, i've mostly only done that for pci passthrough but not sriov
14:58:05 openstackgerrit Rodrigo Barbieri proposed openstack/nova stable/stein: Allow tap interface with multiqueue https://review.opendev.org/757886
14:59:15 sean-k-mooney mnaser: if its a two port card both port will need to be bound to vfio-pci
14:59:55 sean-k-mooney libvirt can somethimes handel this for you but it gets complicated when you have multipel ports on teh same card
15:00:19 sean-k-mooney in somecase each physical port will get its own iommu group
15:00:26 sean-k-mooney on oder hardware it does not
15:00:35 sean-k-mooney older
15:14:03 mnaser sean-k-mooney: i think my issue was thta i should have been on https://docs.openstack.org/neutron/ussuri/admin/config-ovs-offload.html
15:21:28 sean-k-mooney are you using hardware offloaed ovs
15:21:34 sean-k-mooney or just sriov with the sriov nic agent
15:21:46 sean-k-mooney you can use it either way
15:25:28 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Don't detach volumes when RescheduledException raised without retry https://review.opendev.org/757894
15:25:28 openstack bug 1899649 in OpenStack Compute (nova) "Volume marked as available after a failure to build" [Undecided,New] https://launchpad.net/bugs/1899649 - Assigned to Lee Yarwood (lyarwood)
15:25:28 openstackgerrit Lee Yarwood proposed openstack/nova master: Add regression test for bug #1899649 https://review.opendev.org/757893
15:25:44 lyarwood stephenfin: ^ if you have time, going to spend some time on reviews of your stuff now btw.
15:25:51 stephenfin lyarwood: sure
15:26:16 stephenfin Reminds me, I've to review a regression fix from gibi too
15:30:05 jrosser has this sort of thing ever come up for nova? https://cloud.google.com/compute/docs/instances/verifying-instance-identity
15:31:07 lyarwood not that I'm aware of but it sounds like a valid feature for us to look into
15:31:27 jrosser i have messed with a small POC for that today and can share it if there is interest
15:32:26 lyarwood jrosser: I'd think there would be ahead of the PTG
15:32:57 lyarwood jrosser: https://etherpad.opendev.org/p/nova-wallaby-ptg - feel free to add it to the list if you have time to discuss it there in a few weeks
15:35:15 sean-k-mooney jrosser: im not sure if that is realted to what nova-join does
15:35:24 sean-k-mooney nova-join is not part of the nova project
15:35:34 jrosser it's sort of like that
15:35:46 sean-k-mooney https://opendev.org/x/novajoin
15:36:14 jrosser thats quite heavyweight though for something kind of simple
15:36:41 sean-k-mooney the google docs seam quite heavy too
15:36:50 sean-k-mooney do you have a simpel example
15:37:02 jrosser yeah, let me paste something
15:38:45 sean-k-mooney adding some form of identidy verifcaiotn or signigure to the instance metadata for apps to check shoudl be doable im just not sure what woudl be required
15:38:59 sean-k-mooney is the instance uuid enough or does it need to be cyptograpicaly secure
15:40:32 sean-k-mooney actully it sound like adding a signiture to the metadata for an isntace might me enough?
15:40:54 sean-k-mooney " Your applications can verify the signature against Google's public Oauth2 certificates to confirm the identity of the instance with which they have established a connection."
15:41:12 openstackgerrit Ghanshyam Mann proposed openstack/nova master: DNM: testing tempest copy-image test https://review.opendev.org/742547
15:41:29 sean-k-mooney if we were to sign the metadata for the instance you could retrive it before connecting and validate its correct
15:41:33 jrosser i did this http://paste.openstack.org/show/798996/
15:41:54 jrosser and i could validate the generated JWT against the public key after retrieving it from the metadata service
15:42:49 sean-k-mooney the publick keey the user provided
15:42:58 sean-k-mooney for the key pair for the instance
15:43:29 jrosser the cloud provider publishes the public key somewhere
15:43:41 sean-k-mooney oh you want to use a common one
15:43:45 jrosser and the private key belongs to the compute service
15:43:47 sean-k-mooney ya
15:43:50 sean-k-mooney that looks resonable
15:44:10 sean-k-mooney its not really intrusive its just anothe pice of metadata
15:44:20 sean-k-mooney and presuably a new config for the key
15:44:37 jrosser i arrived at this looking at automated cert issuance
15:45:02 jrosser there are tools which will cut certs based on validating instance identity documents
15:45:03 sean-k-mooney lets-encrypt-style
15:45:28 sean-k-mooney we have pyjwt in the upper-constriats file too
15:45:37 mnaser sean-k-mooney: hardware offloaded ovs is the goal
15:45:50 sean-k-mooney so while this might be a new reqruirement for nova it not a new one for openstack
15:46:08 jrosser https://smallstep.com/blog/embarrassingly-easy-certificates-on-aws-azure-gcp/
15:46:19 sean-k-mooney jrosser: so ya i hitnk you sould almost do this as a specless blueprint if you bring it up at the ptg with that example
15:46:40 jrosser ok, that would be cool
15:47:10 sean-k-mooney oh what is smallstep
15:47:15 jrosser was wanting to gauge what folk thought, as i could do something similar with the dynamic vendor data but that didnt really feel like the right tool
15:47:26 sean-k-mooney jrosser:we might ask for a spec just an fyi
15:47:43 sean-k-mooney but as presented this look almost trivaial enough to be a specless blueprint
15:47:54 jrosser smallstep is quite neat actually, i use it here for a private ACME pki
15:48:15 sean-k-mooney ill have to look into it
15:48:22 sean-k-mooney certs have always been a pain
15:48:54 sean-k-mooney to the point where i deployed k8s just to act as a cert ahority and allow me to expose things with lets encyrpt certs in the past
15:49:25 sean-k-mooney now im using traefik for that
15:50:06 sean-k-mooney which reminds me i have to update my openstack redirect so that it force http to https
16:22:30 stephenfin lyarwood: done
16:22:46 stephenfin lyarwood: some nits but nothing blocking
17:31:32 mnaser sean-k-mooney: so far i have made it to the conclusion that my issue is that both vfs and pfs are all in the same iommu group, therefore is not possible to attach a single vf..
20:46:33 openstackgerrit Merged openstack/nova master: Fix a hacking test https://review.opendev.org/754713
#openstack-nova - 2020-10-14
04:47:08 openstackgerrit Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175
08:00:28 lyarwood stephenfin: ^ Morning, sorry borked the unit test change with that fix.
08:00:28 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Don't detach volumes when RescheduledException raised without retry https://review.opendev.org/757894
08:00:28 openstack bug 1899649 in OpenStack Compute (nova) "Volume marked as available after a failure to build" [Undecided,In progress] https://launchpad.net/bugs/1899649 - Assigned to Lee Yarwood (lyarwood)
08:00:28 openstackgerrit Lee Yarwood proposed openstack/nova master: Add regression test for bug #1899649 https://review.opendev.org/757893
08:00:49 lyarwood stephenfin: would you also mind taking a look at https://review.opendev.org/#/c/754695/ today? Another super trivial change.
08:55:39 lyarwood Does the following trace ring any bells for anyone?
08:56:18 lyarwood 2020-10-08 00:40:06.508 8 ERROR oslo_messaging.rpc.server [req-d1269d5e-18c3-410b-87f1-b36a71ec79b9 c6015c4d46694de2bb99411a5fdb388e 301e3b6a1fc941f2897d00a657a22482 - default default] Exception during message handling: UnexpectedTaskStateError_Remote: Conflict updating instance 558b87b7-dbb4-4a3a-9a63-8d873dc4bf33. Expected: {'task_state': [u'migrating']}. Actual: {'task_state': None}
08:56:28 lyarwood during post_live_migration_at_destination
08:57:23 lyarwood on stable/queens
08:58:32 stephenfin lyarwood: not me, but sounds like a race with something else setting task_state. Possibly multiple instance operations happening at the same time and poor locking in place?
09:00:28 lyarwood stephenfin: yeah there's no overlapping instance actions that I can see, just looks like poor error handling, was hoping someone had seen it
09:04:20 tosky as you are around, may I ask how it's going with the zuul v3 jobs?
09:05:15 lyarwood tosky: I've not made any progress with https://review.opendev.org/#/c/711625/
09:05:24 lyarwood tosky: slightly slammed downstream this week
09:05:49 tosky you tell me
09:09:59 aarents lyarwood: I know that nova-compute restart on source host with live-migration will reset task_state, no other idea..
09:13:06 lyarwood ouch

Earlier   Later