Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-16
14:51:30 kosamara Hi mriedem, I've updated https://review.openstack.org/#/c/555861/ Can you take a look if you have the time?
14:52:28 mriedem i'll put it in the browser tab queue
15:13:41 kashyap Hi folks, wonder if anyone has time to look at why is 'mox' raising UnexpectedMethodCallError with this unit test: http://paste.openstack.org/show/719304/
15:14:20 kashyap I posted two code fragments the test points to (from guest.py & driver.py in nova/libvirt)
15:18:17 kashyap mdbooth: Wonder if you have a minute to look at the above (read prior 3 lines of scroll for context)? Last I recall, you made the change to that surrounding code. :-)
15:19:33 kashyap mdbooth: So, that UnexpectedMethodCallError is raised when "if a method is called with incorrect parameters, or out of the specified order".
15:19:46 kashyap Neither is the case, from my reading.
15:20:07 mdbooth kashyap: What have you changed?
15:20:37 mdbooth kashyap: pastebin your local diff
15:20:48 kashyap Fixing the fall out of bumping versions: https://review.openstack.org/#/c/558783/
15:21:18 mdbooth kashyap: I'll fire it up locally
15:21:31 kashyap mdbooth: Then, wait -- let me post the merge-conflict fixed series
15:22:40 mdbooth I can't pull it right now
15:23:36 kashyap Yeah, it's merge conflict; I fixed it this morning locally. Pushing the new rev in a couple. I'll notify
15:23:51 efried kashyap: Note that takashin is doing a whole slew of patches to remove mox.
15:24:13 kashyap efried: So, he'll be replacing it w/ Mock, is it?
15:24:36 efried kashyap: Yes. There's a bp for that, so I'm sure he would welcome the help if you'd like to propose a "fix" for this problem that happens to swap out mox for mock.
15:25:09 efried kashyap: https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/mox-removal
15:25:29 kashyap efried: I'll see about it. Python unit tests is one my Python weakness area
15:25:47 kashyap First trying to remove all the now-needless compat code due to the version bumps
15:26:10 efried kashyap: No pressure :)
15:27:39 kashyap Thanks for the (non-null) pointer
15:28:28 mdbooth efried kashyap: I'd recommend against mixing those 2 changes, though.
15:28:41 kashyap mdbooth: Yeah, I'm strictly following the "One logical change per patch"
15:28:43 kashyap No madness
15:28:58 kashyap But appreciate the reminder :-)
15:29:28 mdbooth_ kashyap: Don't know if ^^^ sent, but I recommend against mixing those 2 changes.
15:29:53 kashyap mdbooth_: Yep, got sent. And then I wrote: < kashyap> mdbooth: Yeah, I'm strictly following the "One logical change per patch" rule
15:30:06 cfriesen mriedem: M0z122: I think this was the last time I tried to push LVM resize/migration upstream: https://review.openstack.org/#/c/337334/
15:31:53 mdbooth_ cfriesen: Aside: do you get a performance benefit from lvm?
15:32:05 openstackgerrit Merged openstack/nova master: Add config drive link to api-guide https://review.openstack.org/561113
15:32:54 cfriesen mdbooth_: last time we checked, I think there was a difference, but I don't know that we've checked in a while. Have you seen more recent benchmarks?
15:33:23 mdbooth_ cfriesen: No, but I vaguely recall that it was marginal at best.
15:33:32 mdbooth_ cfriesen: Over raw, that is.
15:33:56 mdbooth_ Just wondering if you had data which showed it's worth the bother.
15:34:00 M0z122 cfriesen: many thks will have a look on it
15:35:03 openstackgerrit Artom Lifshitz proposed openstack/nova master: Add DifferentHostFilter to enabled_filters default https://review.openstack.org/561651
15:37:12 mdbooth_ cfriesen: Intuitively, I'd expect that if you fallocate a raw file which isn't fragmented and open it O_DIRECT, all of which I think we do, the performance should be the same as LVM, as the kernel is basically doing the same in both cases.
15:38:41 mdbooth_ cfriesen: Numbers beat intuition of course, though :)
15:52:57 Nisha_Agarwal Hi team, I have 2 stable pike devstack setups- one stacked last week on a freshly imaged system and another stacked around a month ago. When i was trying "nova boot" for ironic nodes on both the setups i land up in this error http://paste.openstack.org/show/719143/ due to which it send the "delete" event to ironic node as soon as it goes to "wait-call-back" state. On debugging we found that it happens because instance's context is set to
15:52:57 Nisha_Agarwal None at some point. We commented out this line https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L824 and the deploy goes thru.
15:55:13 dansmith melwitt: did you see my comment on that bottom certs patch? I think she reverted the changes she made for you
15:57:07 openstackgerrit Artom Lifshitz proposed openstack/nova master: Add (Same|Different)HostFilter to defaults https://review.openstack.org/561651
15:57:12 Nisha_Agarwal dansmith, could you help me with above? Should i go ahead and open the defect against Nova for the OrphanedObject error reported on stable/pike?
15:57:59 Nisha_Agarwal or there is some configuration in the config file which i should be setting so that the issue is not hit?
16:01:57 dansmith Nisha_Agarwal: orphaned objects are those without a context set, which prevents them from doing things like loading or saving from the database
16:02:04 dansmith Nisha_Agarwal: obviously no config option should cause/prevent that
16:03:46 cfriesen mriedem: we're running into problems with the code that adds a uuid to services on the fly. it seems that calling service.save() on a deleted service doesn't work. it chokes down in service_get()
16:04:04 mriedem cfriesen: where do we save() a deleted service?
16:04:20 cfriesen Service._from_db_object(), if there's no uuid
16:04:35 mriedem might be a bug that melwitt has recently fixed
16:05:32 mriedem cfriesen: https://review.openstack.org/#/c/555923/
16:05:39 mriedem is that it?
16:06:04 dansmith yeah
16:06:06 dansmith was going to say
16:06:53 mriedem if it's not that, then i will need to see a stacktrace rather than a dead cat
16:07:35 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
16:07:36 openstackgerrit Jackie Truong proposed openstack/nova master: Plumb trusted_certs through libvirt driver image paths https://review.openstack.org/561262
16:07:37 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
16:07:39 openstackgerrit Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158
16:07:49 cfriesen mriedem: so previously we were hitting the bug that commit was fixing...so we applied that fix and hit this other problem. Stacktrace at http://paste.openstack.org/show/719308/
16:08:04 Nisha_Agarwal dansmith, the issue is seen on both the stable pike setups as soon as nova calls this https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L1102 which in turn hits https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L452
16:08:16 mriedem gibi: welcome back - you might have an opinion on the last comment i left here about notifications https://review.openstack.org/#/c/486204/
16:08:24 dansmith cfriesen: is that service deleted?
16:08:27 mriedem i'll throw it in the notification meeting
16:08:34 cfriesen dansmith: yes
16:08:51 dansmith cfriesen: that's why, but I thought we determined it shouldn't even be processing deleted services.. did you tweak something in there?
16:09:35 cfriesen dansmith: the sequence of events was: boot instance on compute node, migrate instance, delete compute node and service, upgrade to pike, create new compute node with same name, migrate instance back to that compute node.
16:09:56 cfriesen dansmith: so now when we look up the service by name, there are two matching entries, one of which is deleted
16:10:24 dansmith cfriesen: but we're using model_query which should filter out the deleted one, IIRC
16:10:28 openstackgerrit Jay Pipes proposed openstack/nova-specs master: Numbered request groups use different providers https://review.openstack.org/560974
16:10:32 mriedem cfriesen: i need more of the stacktrace
16:10:37 mriedem not the middle of it
16:11:10 stephenfin bauzas: Want to weigh in on this comment? https://review.openstack.org/#/c/379525/9/nova/scheduler/weights/cpu.py
16:11:17 cfriesen mriedem: will try to get it.
16:11:24 stephenfin bauzas: We don't do this for RAM or disk. Should we?
16:12:36 mriedem stephenfin: the core filter does
16:12:37 mriedem vcpus_total = host_state.vcpus_total * cpu_allocation_ratio
16:13:56 mriedem stephenfin: also, it looks like the HostState object takes into account reserved_host_cpus?
16:14:21 cfriesen mriedem: I think the core/ram/disk filters account for allocation ratio (though disk is wonky), but maybe the weighers don't?
16:15:08 stephenfin mriedem: *don't take into account?
16:15:42 mriedem yeah doens't take into account
16:15:52 mriedem the inventory on the resource provider for the compute node in placement accounts for reserved values
16:16:03 mriedem but i don't see that on the HostState object or it's wrapped compute node
16:16:18 stephenfin mriedem: Line 1136 of nova/compute/resource_tracker.py ?
16:16:24 cfriesen looks like for ram and disk they just weigh them based on the "free" amount
16:16:26 stephenfin (_update_usage_from_instances function)
16:16:50 stephenfin So I'm guessing it gets it from the ComputeNode object
16:17:52 mriedem stephenfin: hmm, yeah, but that's weird.
16:17:56 mriedem cn.vcpus_used = CONF.reserved_host_cpus
16:18:04 mriedem that's not accounting for the number of instances running on the ohst
16:18:05 mriedem *host
16:18:59 stephenfin mriedem: We do that at the end of the function with the call to '_update_usage_from_instance', I suspect
16:19:12 stephenfin Which presumably appends rather than overwrites
16:19:27 stephenfin cfriesen: Hmm, so as we don't have a "free"-style attribute for CPUs I guess they're correct and we've to figure that out ourselves
16:19:34 stephenfin Or add such an attribute
16:20:51 mriedem stephenfin: yeah looks like _update_usage()
16:21:11 mriedem which also takes into account overhead from the virt driver
16:21:11 mriedem vcpus_usage += overhead.get('vcpus', 0)
16:23:22 mriedem ok left some comments in your change for reference

Earlier   Later