| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-26 | |||
| 20:49:05 | edleafe | Sundar: is 1400 UTC a good time for you? We have the scheduler subteam meeting then | |
| 20:49:13 | edleafe | Sundar: on Mondays | |
| 20:50:15 | melwitt | nova meeting in 10 minutes | |
| 20:50:30 | Sundar | edleafe: Sure, I can make it at 7 am PDT Monday (I am in SF Bay Area). | |
| 20:50:46 | takashin | melwitt: Yes. | |
| 20:51:19 | Sundar | edleafe: Also, this series does not seem to address nested RPs + granular resource requests. Where is that tracked? | |
| 20:53:10 | edleafe | Sundar: hmmm... efried, are you around? | |
| 20:53:24 | melwitt | takashin: okay, cool. if you could send a response to the mailing list, it would be helpful | |
| 20:53:31 | edleafe | Sundar: https://wiki.openstack.org/wiki/Meetings/NovaScheduler | |
| 20:53:48 | takashin | melwitt: okay. | |
| 20:53:54 | melwitt | thanks | |
| 20:59:25 | Sundar | edleafe: Thanks! | |
| 21:02:37 | efried | edleafe: sup | |
| 21:03:43 | edleafe | Can you shed some light on Sundar's question above about nested RPs and granular requests? | |
| 21:05:34 | efried | Sundar: https://review.openstack.org/#/c/554529/ is the first step in getting nested to work in GET /allocation_candidates. Without it, nested resource providers aren't usable. | |
| 21:06:11 | efried | Sundar: Granular will happen here: https://review.openstack.org/#/q/topic:bp/granular-resource-requests+status:open | |
| 21:06:59 | efried | Sundar: There are some who think granular should wait until nested is done. I don't necessarily agree, but we'll see how it shakes out. | |
| 21:07:25 | Sundar | efried: Got it. Thanks! | |
| 21:12:55 | openstackgerrit | Lance Bragstad proposed openstack/nova master: Deprecate rule:context_is_admin https://review.openstack.org/564349 | |
| 21:16:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: General policy sample file for placement https://review.openstack.org/524425 | |
| 21:17:07 | openstackgerrit | Lance Bragstad proposed openstack/nova master: Deprecate rule:context_is_admin https://review.openstack.org/564349 | |
| 21:17:43 | lbragstad | mriedem: two tests are still failing and i'm not quite sure why, it seems unrelated but that's probably not the case ^ | |
| 21:17:59 | lbragstad | otherwise - i attempted to simplify it a bit more | |
| 21:18:24 | Sundar | edleafe, efried: Apparently there is a convention/agreement that only the entity that created an RP can publish traits on it (apart from administrators). If so, Cyborg cannot publish traits on the compute node RP, as Nova created it. Can you throw some light on this? | |
| 21:18:37 | lbragstad | i think nova was just using rule:context_is_admin to set is_admin on the context object (which makes sense given the name) | |
| 21:18:49 | lbragstad | so - in order to deprecate it, that has to live somewhere | |
| 21:18:56 | openstackgerrit | Adam Harwell proposed openstack/nova stable/pike: Add support for listing hosts in cellv2 https://review.openstack.org/563797 | |
| 21:18:57 | mriedem | lbragstad: ok, i've made progress on the granular placement policy patch, except it breaks all placement-related tests now... | |
| 21:19:07 | lbragstad | so i put it in the context object itself | |
| 21:19:26 | lbragstad | and i have it checking for roles set by keystonemiddleware, which should hopefully make things more clear | |
| 21:20:20 | efried | Sundar: That convention/agreement was reversed on Friday in Dublin. | |
| 21:20:48 | efried | Sundar: You can put traits on the compute node RP if you want, as long as they're not traits that compute thinks it "owns". | |
| 21:21:10 | efried | Sundar: but... why would you want to do that? I would have expected you to want to put traits on the child providers representing your devices. | |
| 21:22:11 | edleafe | efried: I think that the virt driver would be the "creator", since it reports its presence | |
| 21:22:22 | edleafe | Cyborg would be an outside agent in this case | |
| 21:23:03 | Sundar | efried: Yes, I'd prefer using nRPs. But edleafe has advised us not to rely upon that in Rocky. So, we are creating a backup plan to apply the traits on the compute node RP instead. That in turn has consequences for supporting multiple device types on the same node. It is getting a bit complicated. | |
| 21:23:14 | efried | edleafe: Yes, but we decided to allow merging of traits supplied by outside parties. | |
| 21:23:57 | efried | Sundar: Yeah, the only way you could reliably support multiple device types on the same provider would be via custom resource classes. And you would then have to unwind that model when nrp is in place. | |
| 21:26:24 | edleafe | efried: that was why I advised Cyborg to wait until NRP was working. | |
| 21:26:58 | edleafe | efried: do you have an educated guess as to how much of NRP will make it into Rocky? | |
| 21:27:44 | efried | edleafe: It's really binary: will nrp-in-alloc-cands make it or not. | |
| 21:28:48 | edleafe | efried: ok, then, rephrasing. Do you think NRP will make it into Rocky? | |
| 21:29:05 | efried | edleafe: I can't answer that. Question for jaypipes | |
| 21:31:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix tox -e docs https://review.openstack.org/564631 | |
| 21:32:21 | arvindn0_ | mriedem: quick clarification. conductor.manager.ComputeTaskManager#rebuild_instance already expects and handles no valid host exception | |
| 21:32:22 | lbragstad | mriedem: added a couple comments to your placement patch | |
| 21:33:58 | mriedem | lbragstad: already replied :) | |
| 21:34:14 | mriedem | arvindn0_: yes, but that's already after nova-api rpc cast to nova-conductor | |
| 21:34:18 | mriedem | and returned a 202 to the user | |
| 21:34:20 | arvindn0_ | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L953 | |
| 21:34:25 | mriedem | arvindn0_: yes i realize | |
| 21:34:49 | mriedem | it's best, imo, if we can fail fast in the API with a 400 - if we're going to do that | |
| 21:34:58 | mriedem | like trying to rebuild a volume-backed instance with a new image | |
| 21:35:22 | arvindn0_ | yea...makes sense | |
| 21:35:30 | mriedem | i don't like the idea of making this an error condition though, because i think that's just technical debt we'll eventually have to fix | |
| 21:35:37 | mriedem | but, like i said, i need to read the rest of the ML thread | |
| 21:36:11 | arvindn0_ | when you say the api do you mean near the API entry point? | |
| 21:36:45 | arvindn0_ | or still in the conductor API conductor.api.ComputeTaskAPI#rebuild_instance | |
| 21:37:07 | arvindn0_ | not sure where the volume backed instance example is | |
| 21:37:30 | mriedem | nova.compute.api.API.rebuild_instance | |
| 21:40:27 | arvindn0_ | found it... | |
| 21:41:42 | arvindn0_ | my main concern is that if we extend image properties in future to allow granular/forbidden etc traits in future...this logic will need to be updated to handle all of those cases | |
| 21:43:37 | arvindn0_ | for granular traits for example, the code we need to add would be as complicated if not more incase we need to handle rebuild taking all those into account | |
| 21:46:53 | jaypipes | mriedem: "the great placement schism of 2018" lol :) | |
| 21:48:07 | mriedem | lance stepped right into a big pile of it | |
| 22:27:53 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use helpers in test_resource_provider (func) https://review.openstack.org/564638 | |
| 22:31:02 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_xenapi.py (1) https://review.openstack.org/564639 | |
| 22:40:29 | openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - Add consumer generation checking logic https://review.openstack.org/564641 | |
| 22:45:45 | melwitt | mriedem: did you mention sort of recently that we had a new dbdeadlock somewhere? | |
| 22:46:59 | openstack | Launchpad bug 1642537 in OpenStack Compute (nova) "finish_resize fails with DBDeadlock on migrations table" [Undecided,Confirmed] | |
| 22:46:59 | mriedem | yeah, but now i don't remember if i opened a bug for it, since i only see https://bugs.launchpad.net/nova/+bug/1642537 | |
| 22:47:06 | melwitt | yeah, same | |
| 22:47:22 | melwitt | logstashin it | |
| 22:49:15 | melwitt | lots in neutron | |
| 22:50:14 | mriedem | neutron has a lot of retry_on_deadlock decorators | |
| 22:50:32 | mriedem | you likely need to add: AND tags:"screen-n-cpu.txt" | |
| 22:50:49 | melwitt | thanks, was about to try that | |
| 22:51:20 | melwitt | [instance: 6dcdb60e-e398-43c0-a6c0-21d0494185c1] RemoteError: Remote error: DBDeadlock (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'UPDATE migrations SET updated_at=%(updated_at)s, status=%(status)s WHERE migrations.id = %(migrations_id)s'] [parameters: {'status': u'post-migrating', 'migrations_id': 4, 'updated_at': datetime.datetime(2018, 4, 26, 21, 16, 31, 790833)}] | |
| 22:52:58 | melwitt | 8 hits in the last 7 days but all on the same change and all in the ceph job. seems weird | |
| 22:53:23 | melwitt | hah, it's my backport https://review.openstack.org/564615 | |
| 22:55:01 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_xenapi.py (2) https://review.openstack.org/564643 | |
| 22:57:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vmops.py https://review.openstack.org/561846 | |
| 23:07:32 | melwitt | how does DBDeadlock work if there's already a @oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True) decorator on the method? does that mean it tried 5 times already and still got deadlocked? | |
| 23:10:51 | mriedem | i think so | |
| 23:11:42 | melwitt | okay. seeing it on an internal bug in queens during the service record report_count update by the conductor service (service heartbeat) | |
| 23:12:08 | melwitt | not sure what could be clashing with that | |
| 23:14:39 | melwitt | although, this is a scale test situation and seeing messaging timeouts, so maybe it's just all of the various service heartbeats being affected by that | |
| 23:17:59 | mriedem | lbragstad: hitting something weird in a test for placement policy; it's a fake token / user context, but it has the 'admin' role in the context, and the rule it's checking against is "role:admin", and these are the policy values from the context: | |
| 23:18:01 | mriedem | {'is_admin_project': True, 'user_id': 'admin', 'service_project_id': None, 'project_domain_id': None, 'service_user_domain_id': None, 'service_user_id': None, 'roles': ['admin'], 'service_project_domain_id': None, 'user_domain_id': None, 'project_id': 'admin', 'service_roles': []} | |
| 23:18:08 | mriedem | is there something obvious i might not be thinking of? | |
| 23:18:16 | mriedem | i figured 'roles': ['admin'] would be enough | |
| 23:19:11 | mriedem | oh sorry, it's actually using this rule: rule:admin_api | |
| 23:19:19 | mriedem | which is defined as: is_admin:True | |
| 23:19:44 | mriedem | and i'm guessing that's failing because the policy creds don't have an is_admin key? | |
| 23:19:49 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_xenapi.py (3) https://review.openstack.org/564645 | |
| 23:20:14 | mriedem | i think this might be due to that goofy context_is_admin middleware thing | |
| 23:20:22 | mriedem | which i'm not using in the placement policy stuff | |
| 23:23:24 | mriedem | ah yes, nova.context.RequestContext overrides to_policy_values to add is_admin https://github.com/openstack/nova/blob/master/nova/context.py#L263 | |
| 23:46:52 | mriedem | yup that was it, got it fixed now | |
| 23:47:18 | mriedem | melwitt: would be sweet if we could build docs locally again https://review.openstack.org/#/c/564631/ | |