| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-17 | |||
| 18:38:10 | dansmith | efried: idk about monkey vs. mock.. I had never seen mockpatch before until you used it | |
| 18:38:22 | artom | mriedem, you mean the thing I said "lemme do that now"? ;) | |
| 18:38:24 | dansmith | melwitt: it just depends on what you're removing | |
| 18:38:29 | mriedem | artom: no, this is different | |
| 18:38:35 | mriedem | artom: another one of your waylaid newton backports | |
| 18:38:56 | artom | But I have so many | |
| 18:39:26 | melwitt | dansmith: for example, the reserve() method passes straight through to the quota engine code and then onto the DB API code. so to prevent breaking the contract, we have to keep all of that no? | |
| 18:39:54 | melwitt | I had thought I could just put "pass" in them. maybe we can since reserve() didn't return anything IIRC | |
| 18:40:07 | dansmith | melwitt: well it depends.. if the side effects of the code are no longer desired, and the older remote can't tell a difference, then it doesn't matter | |
| 18:40:10 | dansmith | that's the point of the isolation | |
| 18:40:31 | dansmith | you're not breaking the contract by changing the behavior of the remotable, only if you change the remotely-visible behavior | |
| 18:40:48 | dansmith | just like we make save() do new things like generate uuids even though older clients don't expect that | |
| 18:40:54 | melwitt | yeah. I'm trying to think if the other side can or can't tell the difference. reserve() currently sets the 'reservations' attribute | |
| 18:40:58 | dansmith | that's the new behavior, we just have to make sure we honor what they're expecting | |
| 18:41:10 | dansmith | but not if NoOp is used right? | |
| 18:41:25 | melwitt | yeah not if NoOp is used | |
| 18:41:37 | artom | mriedem, ah, OK, the rebase thing. So your fix should merge before we merge the backport, which would squash your fix | |
| 18:41:45 | melwitt | so it should already be resilient to that | |
| 18:41:45 | mriedem | artom: yes | |
| 18:41:48 | dansmith | so presumably we can keep the interface stable but neuter the new-side code so it's not doing anything right? | |
| 18:41:50 | artom | Honestly, we can do internal backport to newton, so no massive rush | |
| 18:41:54 | gregwork | artom: http://www.mellanox.com/related-docs/prod_software/MLNX_DPDK_Quick_Start_Guide_v16.11_1.5.pdf | |
| 18:42:04 | artom | But upstream would obviously be appreciated :) | |
| 18:42:12 | gregwork | artom: section 3.2 is the recommended settings for starting a kvm guest | |
| 18:42:18 | melwitt | dansmith: yeah. looks like we can safely do that for any method overloaded by NoOp since NoOp does a pass for those | |
| 18:42:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/newton: Catch InstanceNotFound exception https://review.openstack.org/467321 | |
| 18:42:34 | dansmith | melwitt: you'd think yeah | |
| 18:42:37 | artom | mriedem, ^^ that's the tenant usage instancenotfound thing | |
| 18:42:58 | artom | mriedem, I guess I can give you the +1 on your fix, but that's about the extent of my helping to push it through | |
| 18:43:28 | melwitt | dansmith: yeah, not guaranteed. some code out there might be assuming only the default driver and thus assume something like 'reservations' attribute would be set. I'm not sure if we need to account for that | |
| 18:43:29 | mriedem | i need a stable hero to review these: https://review.openstack.org/#/c/493206/ https://review.openstack.org/#/c/506323/ https://review.openstack.org/#/c/509441/ https://review.openstack.org/#/c/508902/ | |
| 18:43:35 | artom | With great power comes great responsibility, with no power you just sorta chill | |
| 18:44:12 | dansmith | melwitt: you can probably just make sure reservations is set to something that will make it not choke right? | |
| 18:44:55 | melwitt | dansmith: yeah, maybe empty list would be good enough. hopefully no one could assume non-empty list | |
| 18:45:15 | artom | gregwork, ah, OK. So they're not saying "in your openstack instance, make sure this command line is present" | |
| 18:45:29 | dansmith | melwitt: I don't have the usage of that field in my head so I'd have to look, but AFAIR, it's just passed back to commit or something | |
| 18:45:37 | artom | gregwork, you just happen to be using openstack, whereas the doc is written for bare kvm/qemu | |
| 18:45:44 | gregwork | artom: right | |
| 18:45:50 | melwitt | commit() used to set reservations = None too, so that's a thing that's considered valid | |
| 18:45:58 | artom | gregwork, nova does do hugepages and CPU pinning | |
| 18:46:27 | melwitt | dansmith: yeah, I think only the object itself uses it. or at least that's the intention. but ppl could access it if they wanted, I don't know why they would tho | |
| 18:46:31 | gregwork | artom: i am getting a qemu error trying to initialize the network with dpdk, i am trying to troubleshoot by making sure qemu-kvm is being invoked correctly | |
| 18:46:51 | artom | gregwork, I dunno what version you're on, for for pike the docs are https://docs.openstack.org/nova/pike/admin/huge-pages.html and https://docs.openstack.org/nova/pike/admin/cpu-topologies.html | |
| 18:47:08 | dansmith | melwitt: fwiw, I think you're overthinking this.. just set it to something and grep around for the uses of .reservations and make sure it'll be happy.. should be easy I think :) | |
| 18:47:34 | mriedem | artom: +2 | |
| 18:47:36 | melwitt | dansmith: I was getting confused by the idea of accounting for code that isn't in nova | |
| 18:47:54 | dansmith | melwitt: eh? no need to do that, like always | |
| 18:48:00 | dansmith | melwitt: what are you thinking of specifically? | |
| 18:48:11 | artom | mriedem, woothanks | |
| 18:48:24 | gregwork | artom: working with ocata atm | |
| 18:48:24 | artom | gregwork, wait, you're getting a qemu error while spawning a VM through nova? | |
| 18:48:28 | artom | That's... bad | |
| 18:48:38 | gregwork | artom: yeah :/ | |
| 18:48:50 | melwitt | dansmith: I thought that was what you were getting at. nothing calls these methods anymore as of queens, so I thought it didn't matter what they return or not. but I knew I couldn't remove them if they were remotable | |
| 18:49:01 | artom | gregwork, and is most likely a nova and/or libvirt bug | |
| 18:49:09 | artom | Because that should never happen | |
| 18:49:49 | dansmith | melwitt: no, I'm saying we should keep the contract until we bump, but not for out-of-nova uses, just in case someone has an ocata compute running by accident or something so we do our best to not just cause it to break in weird ways | |
| 18:49:50 | artom | Well, assuming the hardware actually supports what you're trying to do, and you've configured nova correctly | |
| 18:49:58 | gregwork | artom: i dont think so, i believe this is related to ovs being configured to use mellanox dpdk poll mode driver, and kvm is not setting up the guest correctly to plumb the networking | |
| 18:50:07 | artom | But even a misconfiguration shouldn't go all the way down to a qemu error | |
| 18:50:22 | dansmith | melwitt: since it's all conductor-side for those actions, just attempt to fake the result that won't cause any of our code in the last few cycles to choke and call it good | |
| 18:50:34 | mriedem | sdague: can i get you to poke through these stable/newton changes? https://review.openstack.org/#/c/493206/ https://review.openstack.org/#/c/506323/ https://review.openstack.org/#/c/509441/ https://review.openstack.org/#/c/508902/ | |
| 18:50:37 | mriedem | tomorrow is newton-eol | |
| 18:50:52 | gregwork | artom: eg. https://paste.fedoraproject.org/paste/GWEv1HyVo1y9sekJbZFBeQ | |
| 18:51:00 | melwitt | dansmith: oh, gotcha. I understand now. I'll go over it again and look for those possibilities and update it. thanks | |
| 18:51:20 | melwitt | I at least can remove the non-remotable one in a new PS | |
| 18:52:00 | artom | gregwork, you're very rapidly bringing my out of my (admittedly limited) comfort zone :/ | |
| 18:52:58 | gregwork | artom: so i am testing vnf's on my openstack, where the compute nodes are running ovs in dpdk mode, with a poll mode driver (the nic is not managed by the kernel) | |
| 18:54:49 | artom | gregwork, sorry I can't be of more help, but I'm out of my league here | |
| 18:55:14 | mriedem | gregwork: sean-k-mooney might be able to help with dpdk questions | |
| 18:55:32 | artom | We have some folks from mellanox who hang out here, but they're (I think) on Israel time | |
| 18:55:42 | mriedem | moshe is yeah | |
| 18:55:43 | sean-k-mooney | mriedem: gregwork sure let me read back | |
| 18:56:31 | sean-k-mooney | gregwork: so yes you have a coumpute node with ovs-dpdk. what is the question? | |
| 18:57:20 | sean-k-mooney | gregwork: ah you dont have hugepages requested for the vm | |
| 18:58:12 | sdague | mriedem: done. I was a little surprised you wanted some of those back to newton, but that's fine | |
| 18:58:35 | gregwork | sean-k-mooney: have you seen that error before ? | |
| 18:58:42 | gregwork | the one i posted in the pastebin | |
| 18:58:49 | mriedem | sdague: the bfv one was needed for another thing someone was reporting in newton but w/o better debug couldn't be determined | |
| 18:59:23 | sdague | mriedem: ok | |
| 18:59:23 | gregwork | sean-k-mooney: im using the mellanox rhosp11 plugin which has configured ofed on my nodes and we are pushing a nutron dpdk config | |
| 18:59:48 | gregwork | sean-k-mooney: the deploy works great, but when i spawn instances i get that error in qemu-kvm on the compute node | |
| 19:00:03 | gregwork | sean-k-mooney: networking doesnt appear to be working (most likely due to that error) | |
| 19:00:14 | gregwork | vm boots and all but is an island | |
| 19:00:15 | sean-k-mooney | gregwork: the one you posted is indicating that the guest memory is not backed by file descriptor backed memory that is mapped shared | |
| 19:00:26 | gregwork | sean-k-mooney: yar | |
| 19:01:26 | sean-k-mooney | vhost user requires file decriptor backed shared memeory that is preallocated and locked to work | |
| 19:01:51 | sean-k-mooney | in ocata that means the guest must use hugepage backed memory or it will boot but have no networking | |
| 19:02:17 | gregwork | what am i missing from a flavor pov? im reading about some keys i need to set | |
| 19:02:47 | sean-k-mooney | yes you need to add hw:mem_page_size=large | |
| 19:03:12 | mriedem | gregwork: https://docs.openstack.org/nova/pike/admin/huge-pages.html | |
| 19:04:17 | dansmith | efried: I assume you're good with this since you pushed the latest edit? https://review.openstack.org/#/c/470575 | |
| 19:04:30 | dansmith | efried: no +1 so just wanted to make sure before I sent it to oblivio | |
| 19:04:33 | dansmith | *oblivion | |
| 19:04:45 | efried | dansmith oblivio away | |
| 19:04:56 | dansmith | efried: are you going to rebase the rest? | |
| 19:05:16 | efried | I could, if they need it. Do they need it? | |
| 19:05:21 | sean-k-mooney | mriedem: sfinucan wrote a section about vhost-user/dpdk also in the docs im trying to find that | |
| 19:05:22 | dansmith | I think so | |
| 19:05:27 | efried | ight. | |